1:02 p.m.
Hi,
We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of events
like these:
2016-04-13 10:52:30,735 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-86) [60dea18f] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com) moved from 'Up'
--> 'Paused'
2016-04-13 10:52:30,815 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call
Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has been
paused.
2016-04-13 10:52:30,898 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call
Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has been
paused due to no Storage space error.
2016-04-13 10:52:52,320 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-38) [] domain
'5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
'host6.domain.com'
2016-04-13 10:52:55,183 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com) moved from
'Paused' --> 'Up'
2016-04-13 10:52:55,318 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, Call
Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has
recovered from paused back to up.
The storage domain is far from being full, though (400+ G available
right now). Could this be related to this other issue [1]? If not, how
could I debug what's going on?
Thanks.
[1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
1:54 p.m.
Hi,
What kind of storage do you have ? (ISCSI,FC,NFS...)
Can you provide the vdsm logs from the host where this VM runs ?
Thanks,
Freddy
On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es> wrote:
Hi,
We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of events like
these:
2016-04-13 10:52:30,735 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-86) [60dea18f] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com) moved from 'Up'
-->
'Paused'
2016-04-13 10:52:30,815 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call
Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has been
paused.
2016-04-13 10:52:30,898 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call
Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has been
paused due to no Storage space error.
2016-04-13 10:52:52,320 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-38) [] domain
'5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds: '
host6.domain.com'
2016-04-13 10:52:55,183 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com) moved from
'Paused'
--> 'Up'
2016-04-13 10:52:55,318 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, Call
Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has recovered
from paused back to up.
The storage domain is far from being full, though (400+ G available right
now). Could this be related to this other issue [1]? If not, how could I
debug what's going on?
Thanks.
[1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
2:04 p.m.
--=_0dfc307e5d88262f38c3a472ce71071c
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8;
format=flowed
Hi Fred,
This is an iSCSI storage. I'm attaching the VDSM logs from the host
where this machine has been running. Should you need any further info,
don't hesitate to ask.
Thanks.
Regards.
El 2016-04-13 11:54, Fred Rolland escribió:
Hi,
What kind of storage do you have ? (ISCSI,FC,NFS...)
Can you provide the vdsm logs from the host where this VM runs ?
Thanks,
Freddy
On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es> wrote:
> Hi,
>
> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
> events like these:
>
> 2016-04-13 10:52:30,735 INFO
> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
> 'Up' --> 'Paused'
> 2016-04-13 10:52:30,815 INFO
>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
> has been paused.
> 2016-04-13 10:52:30,898 ERROR
>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
> has been paused due to no Storage space error.
> 2016-04-13 10:52:52,320 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-38) [] domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
> 'host6.domain.com [2]'
> 2016-04-13 10:52:55,183 INFO
> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
> 'Paused' --> 'Up'
> 2016-04-13 10:52:55,318 INFO
>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
> has recovered from paused back to up.
>
> The storage domain is far from being full, though (400+ G available
> right now). Could this be related to this other issue [1]? If not,
> how could I debug what's going on?
>
> Thanks.
>
> [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
> [3]
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users [4]
Links:
------
[1] http://vm.domain.com
[2] http://host6.domain.com
[3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
[4] http://lists.ovirt.org/mailman/listinfo/users
--=_0dfc307e5d88262f38c3a472ce71071c
Content-Transfer-Encoding: base64
Content-Type: application/x-gzip;
name=vdsm.log.gz
Content-Disposition: attachment;
filename=vdsm.log.gz;
size=27265
H4sICB8nDlcAA3Zkc20ubG9nAOxdW28bR5Z+n1/Rb7IQFVX3C3c8wMSeWQTYTAI7XsxDAE1dbW4o
kkNSTmYH/u97ipRM0SKb1d28aWPAkMluNvs7dU6dy1en2D99mEYbEOWq33/9l2/f/We/TzGRCHNE
WEVwX5A+xleKin7fDcf+l7ev+31Oab//dj6e2vex9/1g5nvxN//qNvT7L97H+Rv4wtdxaP91WV3f
zabXbjC6ntvZL7M4rxDykzs0HMzmFUaUrT4QQjVIL69D/HjtCHMmWo9IohFxqyJyPhjETRJJC+FZ
0Ne3cW6DnVu4amjfvwyDafTzanz/FaO74bBys5ccG1n58d1o/pJUL/yvofrbeBQv//BTodQCt5H6
7btXr/7y9m2/+mOcTv9UvawuyDe4AoDjaZhVg9HPo8fvx3fzn0cLoO5f8zirXvAeqX759hJwTwYx
XFW4hzEhRlBZza4q1uPV999ez34eXfxH9cepz9+PVwLxGoH4lcLHU6MIkVuMMbLGc8S18UjHGBHB
TGrJleeE7UuNO6Sm+lzUiIVmbKlGs0GN/zMbj6YT3/vLb9HfAcRrWi+YAHVmnfT7woiVVD/Boe/t
CF5OF69BtJu7CYxzfDuHP5dVPvjyHywJ7gV8LZU4IM49Rk55gUyMLjhiQgzyH/3+7fjjYPS+StPx
bTXL14PsA7CBP92/m0zjxE7hIxvhf/e3v/5Qg344fv9uPhjOQC+gpDCYTezcf4hTgPzr1E4mcXpZ
vbkbVXYU4EbjOZhHH8Z8Ms6SzF6MJ/PBeDR7ubSHTveHg43vf1W9ifO76QiOzCaAI/arf99d8ESk
gEFEUlALbsthZLRjKHDnPU7GEEYv4JMXfhwivMBX1YX18zs7hDc/Te8ivP8YpzOQDA6wxdl/3sHU
CKvzIdsovL3IZoWpFFpcwOGhnc1ffYj+l3xK9Uw+9tEOBw9Xfrqq7i6UcgHrPCBagd5lsMhQKpEP
1FIvopU2luPDBfiYYNjoJwBFT20EWOI69jqAy3n5BB+c2oivJELtF9/C/T/Bx7fg8zgmSzwBixMC
cac0csFiFIhSURNHNAl7VjDDmBcZ4KfmTk4+ODkYBlLr5ZauqKmDS+DPZh9i+Dp5v07er5P3UJP3
qBnK55xklaY8zPJWIkB8H99NfbzH3O8bzlbSvFk/2/vh19Eig5jGYbSz+Ofh8LJaHOutjkDK8M+7
OJvPqn9/qh6+Pr/ZEz6Nd+PzduTj8BG8zwceo+ugc2NYvc5D9NOYGqobrqhwZd14Os8a/qsdzuLn
WiCXbFvBySutZXb4w5hLAaLYgQqg6QcoX3Jhg3wczeP0+nY0v84vpwAWbtgL41s7GPX8+LZ/A0ns
TfxtAuJcl0z+a7j25jbsq3YiteNlCDnL8bodvJ9aDz7uuiRg73fIdpiYYW2G7Gm1ib8ha9Xm4/eL
apNx/FBs5pdf1JqYcS5NrjWNllCJbqUMdhgA48eShq+k4U+lUYzrhTSU1ElT55vMFTbmLM15MBvP
rktSuz1P/vrRIkQdSfeMfdY9vPxS9zTbssjKJz1Zx37V0Hkk03nk90ZiZqnV2ZCYWDGhluyX3qBG
uLsHZNV8IdhW6obwKyHAFjMdM/bjYYiZlRlDQqNAth/vj76+P9r7s/dxAi++u50MQcoPdhSG8cYu
jl5Wy7M5jfDj0QiugJCyTCQJVT1A3SN9zhnlT9BtHfIMT2+CpzfBe/359Iubm0ys3dxcVu9mGVHO
wXLefjMb/G98SUijATKbEBCi6yHcj07++stqeTwuia98RfXb7XAPgwPQYLYEEBG+DxJLwEUNGPHf
b4dPoczAcCMoKt96DHVN9WE+n1ThAdoCzYuLz3igZFkguvzDt8tb/P37/3rz46udY/WAJDu5j2HW
W172Nk4/xrVxWWTFMDvmdpl83h+plh+YVgkG4cvBuZ+ohHJuZC0SiVdINN+I5AaUAYXC7AHL7LJ6
sxtDroGmMF7lWCThKyyG7hfLeDJZYaml7wGJYsdz22eyaJGlzm79PNw20UbvXnqqy4pJziPFWWZe
Z1l45fES51monmvhlYcsj9IZFF5UYSn0Ml1ldenqDgtQbSxg/5UXNeAC7rNvWidOTTFBoJig+CwN
+gxLrzxardzl/ksvRo3GZql8XLJiXVOMUAxGoLvwwYrpiC0jKLlEERdRIpMgRtMkjOJBRCx8txVr
tTUnekB/2BXrJvd/1ivWC6fyZNGB9ei5rHotDP8JQLJl1eb4q145K9uwKMfPZNFrUes/wWe24Dv+
otcySBcZ4IYVkGIv13TJutDDnXrJ+uvs/Tp7///OXnOSHKXRmnWBCCdds26F75Br1sU6b7hmXaju
2jXrOgafZgaf/t7WLbLU5ozWLSgz29ctShg9ytt2kz9nHjNLLc+n+VpqWrP8VMIy0ba0zO+UyMzj
ZfRZjte5EplUXWFytJ6LWiKTY8HMYteJBv/XroVkIc6x2gh2EJlKMtKNx6SZmTtPez5DHjOPljwW
J1/PY3IJ+cRC+YrQDaZcuLQOEsm8Fe6ovQeieHk9w2P0lL0H2xEcqPeg+eAcuvdAlPYeLACxQ/Ye
iLX1/hpu+Qsk++w9EBt7D+qx5CbdQ/QeiKe9BzVFCMtFyO+uZQyk1oSdT85Oxfb9kpM4HYzDwF8T
XJOCMHKFs3W353ScDJBDc8gQFSSHHNJFBFmiQF4xF1SyGLLqRutO68C3zIQH3K1WnEA1/z0e3t3G
t+DAX8zCu3ffvX5ZyHReVbPJ/eeVcEknrpFROCFOHXxeGImStk7G5KiT+fOD2/f3FzDLRaDBIUxx
tuIEGQiWHlHiHOGaROPzBR/Hw/sLtLfwtcyiRC2BsfUYaWI1SgxupuEtp/mC9XWz8vHjLVfM1sZv
06rZxXwK3gZOLohlwyURCgu5IJjhS+0UcrVNpz81sVredh2h0GIfrSMcV56jzsI6ZrUZ+GNyql2R
7ZFNbaPhhjxqoXI38qiP4NWQUQt4nTbMWx1l4paB90sOcZwE0gxitWKGBq6MVbZ1GOD1bqztVvlT
hQHFuDEJbuCkBq/OHEGOEIWkJCll7UoR1sKAsF5rIhhkPzkMCEaR5TDK1OsgUlKBJl0XBpqM3wHD
ABbaQJmr5Ga3uTr9qZXVNg0DhRa7PQwcWJ6jzsLCMFAA/kRhoBWyw4SBXVDabgEtVG59GMid7WXo
2tgf94YbTqFOCwYiFQ0KGQsZrsKYQnqbsHKpXRSo6ch/gP2sgkDwSimeoOKVeS2CgB4d8QEZamSO
64I7vBYEOMM+ckJQcBwKXwLpgCacIWmiD5JpzYTZHgR2DZ86Sgzg+QcdGNF8Y+b86OynBgar2gaA
QmPdFgAOK8xRZ1+R9y/BfhLn3w7YIXx/uXobuv5Cze6qAHYVKJ0akJ1XhCXmUWQisxvaIku5RA5j
Yp1mSdFwICKobevxqZx/opgTqxwiTkjEA/xxHhukQeuCeuqc9GvOn6YkOAGtOx0tlAyRw7BCtIiB
isSh/mMKdyCC2rZON60AGFVMUEhaNmfMq9ONyurWDaWFFru9AjiwPEedhXsigvTZEkEbkZ2GCNJt
iaAy5XYlgjr1eDpDIZk1BPymzs7QgjNk1qJgSVIs0BCdOBARZJ5ZGKDEhMA01ADKUMSVjMgy73Nb
gnXYCiIWfdqrMABZvo5YJWQlfDmnmCGLk0TExghlBLbJ1oaBJuN3uDCgDdisMZLrjW7z0elGZbVp
HQbKLHZrGDiMPByfZBbuhwjK4M+TCNqM7CRE0ErDTcNAmXK7EUHd7I+rRISLAgnPIFJhpZdsBddc
AeiEwRkehAjKsJ9VEIiYMaEDQ0IZGB+pOPh/JeB+XAYvIb5/sRqgKAnYwPdKHBXUAgQj4yxFgtJA
LXeYhZogsGv4yFFigKCMc0OY2Zg5PzrboLjO0FsSQWXGui0AHFaYo86+vRBBGftZEkGbgZ2CCFqp
tykRVKbZjkRQR+ujzgkXKHIW/BiPwkCVgh3UKz4kJZNSsqXv30Vk5IbbfTr/os2FnVYBEpfaS4mY
plABBOqQhfCCNMbaCi7g+9dXASh8mkYnkRK5AvQiIGPhhlp4LqAIIMHw9kTQ+vgdzPszCvkLFVxt
dJiPzjYpqjP2lu6/zF63uf8DS3PUGbgfEiiDP08SaDOyk5BAKw03DQFlyu1IAvFOTaHBWhldxAj8
pAVk4OFc8gGcm6VMEsscc4chgfhzawq1ITCo2DhSBMThwsIoJcIRI854hyO2fn0tIFOAFOoEFAED
hCIlkZaYIM+ZkMwl5zRtTwKtj9/BQgAlXHHNtnAmj842KqhZ2xBQaK/bQsCBpTnqDNwTAcTOlgA6
YkNoAZRWIaBQuR0JoE4NydlFUYk1Cpbl3XYkMxpJIGkzQUGwY8oehgDizywAYGwho08JPqYhAGCe
kOFwKxGpZt5qrLxYCwCQ4hMoDRSSGG4DVQlDjmoCsDC88TIGU7MrYNfwHakbFCumONGUb+TMV2eb
FNWtW0ELjXXrCsBBhTnq7NsPAXSqNtB2wE5CAImWrr9Qs10JoE7WpxlRxquEqKO5tzEapE30COKU
SC4ZBz7tQATQc9sLEIhK0SaWfTlcoBVHlmmHcAiCJ/AbwdI1529UspxTgrjTkP07uMpxLlCE1F8S
QgTjHbaEcfmcO4F466ehFVrskTuBeKde7NazcE8k0JEfDdUV2WlIoLZ7AQqV25UE6tSOrLDW3EQN
3jJvdyVa56dVeeRJtEp4zi05UCcQV88sDGCvvMfcIYe5Rxw0i5z3FBlMiaFWq0jcF+sALsYEESAy
D7FAYoWsIvmnjLwH7RvnbRcS6EjbATg1RAu8OW9+dLZRUd16P0ChvW7dD3BYaY46A/dEAp1qR0BL
ZKchgdruCShUbkcSqFMzMlPERUk8QIwSvJThSFOoXYgMkkcrnDXxMCTQs9sR4HWUziQkDEQBLkjI
ykyI6JTA1xsC16wFgLztF3MWUaSEIu6jQsbYnAA4Y6FoEEy13w7Gz3lDQLnJNn4mcpm57rcKONMZ
uB8i6FS7AdoBOwkR1PbHdQs125UI6rQXQJi8ZdlrpHHKv18jIMGNeduy8MYJLMAd4gMRQc9tL4DO
671JwgUYvDj3HEZJRY5opJwmpplK651AREvNfGCIRQVjG4JAzmCBkuWU5hTAKNWBCDrOXgChFGGS
MrwxZ350tlFR3XonQKG9bm0EPYQ0olMfdusZuB8SSJxuH0ArZCchgUTbfQCFyu1IAnUzQA1lCouQ
nTLJILENPCAbwWtZFhmBJDX/es1hSCDx3HYCYI9V4C53cqXcY0oxqDMSGC+uJOMsmS9CQNBWgv49
+HwNV2lq4F4sIEdxMjgabBbbiFuSQOI4WwHatU/ustnWewEK7XW/zaDF0hx1Bu6HBBKn2g3QEtlJ
SKCVhpuuA5QptxsJ1M3+vKDg7PIStYYclQMmZKQEN6owAWkJV/EwnUDrDuwZBAAYpDwmCYHSIJ5T
HZHmgSIfPRVgQ0YvthusAoAUlHOWl4BjvkAQgYz3DGpAQxm3VglREwB2Dd9xNgMconlGtN4LUGis
x+wEWglz1Nm3FwJInGonQDtgpyCAVupt6PoLNduRAOpmfVoH78DmkaA2u0ADKa0mBimPIThhZa1j
hyGAxJ73ARx8K5jghAcmLQo+/xiEtwQ5lTjyWCsf4NuVXd8HHBJWJEX4LGES7gB+XwuLEbVUWael
Ncy2J4DWx++ASwAH+TFl0XonQKHFHvnHoVfyHHUW7okEOt1egFbITkMCsbYVQJlyu5JAnTYDROwl
EwIjH5SHzBleGaopojIoRbxIgjZ7NnU5ifHcdgMQwwyJiSHh8p4+qiWyJGIkg8+PBlHKE7teA1Aa
BQVEwVGMOPcKij9qUPRYeattkoun0bYlgY71jIB2DZTFVts0DBRa7L4bQs9zFu6JCDrdMwJaITsN
EdR2S1ihcneFAbYDXqd9AQKw4ZATWwhNedsauCoqNFJGeRVoMMbqtmGA1bux57YvAGsbfRQKMYGh
yEgUkvscBkCVOGriA/iP9V3BCUfiHXwiZf17GFsTo0HYkUA5ptGIWBcGmozfWXFBDWy28XJwmb3u
kww61xlYGAIKwJ8oBOxCdsQ9AQVQ2i0Hlym3/nHbdfGJH/Vx22fynOYs9fk8bpuyh6dtsw3PfHvy
ZPe6olNcMd3JmZCkLCVQajhnY/4FqoC01wYZ4XgyBPSimv3Ixyb424LSPfpWQR1uOM6SzF6sh8NO
928QFD/ff1NAvLsoeebyRQ42fhxyKME5uvj5nR3Cm58gAMH7j3E6A8ngAFucXT6xdHU+ZBPNYWj1
9N8cpIZ2Nn/1Ifpf8indWwSuj3Y4eLjy01UFqUbBg3TL8eECfMsn1D4BKHt0I8Ci7GyfA7iYlk/g
4Z7cCK+ISt4rvPykyCfw+BZ4JU9J37N6F8/SLrK/Tx18XNP0r9C/PUr/vs7dr3P369w9xNw9an5S
V+y0EeGYJc9+8O2x8GkMqG35U6ju2vIn94xuBaeuMFHZ3w9jrgOIYgeqfqYfoHbJVQ3ycTSP0+vb
0fw6v5wCWLhhL4xv7WDU8+Pb/g2ksDfxtwmIc10y+6/h2pvbsK/CqU6ZavnYvTMcr9vB+6n14OSu
S+L1fodsl4kJ3WLInpaa+BuyVmo+fr8oNRnHD5VmfvlFoYkZVlosSk0CdejTUrPUAiQ5ljh8JQ7f
JI5kskAcWiOOuQJjyQED6tcxuEBG+CN5Ho6+uAEXAy71w+uF1UH9t3ybvc3SEKuSLOwBEqGcG70b
1c1NLqtvbgAWBRP77ocfp2OICbNXwwHMhAWsOwADEXLp+D5/oNfrrd3L1N9Lk73dS+C6uQD3yr1T
+7jXLrVC2HhkpUKttHr//1KVEH8GCeJavuk9qqxdGx6pFv5N7PxD1dwpDWbj2XUTy9gtFUy3Sc7m
ZnO47+uB/z/2rrW3bWNpfz+/gjhfnBSisncujbpA6rit0Tg27PRyUOfV4TXRqSwZkuwqOD3//Z0h
KVuSSWlJ3eiEAYpavM7Mzg5nnpndAcqpO4NeXSycnGHrGF4xjkLLsx6fYIVwlfVXF9j7AYR1ltnD
y98s3wv+jPqhMWX4XVikjHGxhLJsTr208AVWr9vHmT5PxcujPw6O376+ujo6HQ3Q+3xzcnV8eXrx
/vT83dHp1fkVHjs9P4cDZyfnv7y/Ojk+okmZztuT11cnlyfvL09Pro74wxG8Di9Ka3nenh//fHH+
9vT4X0fTn5cn705+e/329N37k8tfX7/FaxNf/eL8/G0nScwYJn4OLk/Ozt+fdC5ev//pqEBJXoGS
vEIlSa4/f3tydBl9vOt5Q/x99SZ9nUl4B5e//9fFydG7HxKJ/HpyeYUSSpjsXP30unP889UvZ0dc
i4DwSAseh1RonzAqPC9kPIq0FkB64AZOxMXBB/NhV4/TTDm0YJrBaHdv4NeP3tCH/x0Pej340A4Q
SYsyZ30Y3fS9PjiYg9iCeCRCTU3uGVl/zJNThN9l1Izg4yMEK6YEjPlHVMTh1A1+591Eo1sPLAzS
kx6z+tODRqa9k5D67sryekjoZ2v6jigsTbskO6b9PknnrSZ+qR7U0402tsAbdqJXSEuyHblQnD+4
UPDnogvFXOI6Bh6UU8yNIC2NPtOOMkgmYM2mxnAF15zWJYMEAY100hSSyhlGeDvaWmucMFZog4Ro
KURTMacxCAa9MBonNvrwEDOEF9nRN9nR9usgiG7hj9Ob2x5w+cnrh72o4yVHX1rpWbTrwaDfhzsg
MkvxGMqcNlDdxhdypp5QVyhyJM/JI0/nkffm4fSMB/TLKP3SpPhXB/P2R5SWElAuBZTq5SRk0sHH
v7TS41GaPcI7rMlNbwPCARpgtqDnCs8b3gZAF0N6f7/pPSVlBIobwUDhqwf30dD6NB7fWuGUtISa
FwcP9IAjkVD08h/fp6/4/ezt5cXxclkBQVNK0Mbdh6N2ettVNLyP5uSSgEszLn92xEovGFoxCGFR
OLNRR3Fab5ESLXIp6dymIcaUluRzupIGhBKH48fP5GpaHCwHm9KCjvsmaRnc3s5+spekwJESxOK+
rsQ/cK2JWxezDYbbFWi2RZss+fouQ5cEwjFkE/iFCdBohBAlJLFNkGQC5JlLSewfDiiB6Zbia0+A
wColcPYv8XKocCnW9iP0FeqQrEnZIgrzPZBzd7sIxJz8fnF++f4LgGLSCboEjDFKxhqCMTIWXIRO
rGlAPEZ5rGL4GFJwITylCeEBh+e7ks6CMcv1ctuj/wZ+L4792XSGfQHDfzPLS4EGGFWDFGgAX9AA
LTn1nZiGQgsCauW7xBNhqMIg1mEYMO7EMhJKzmjAyvnPdgrHITlFvm5GzcbhOJM5aALHmdK+STjO
iHYjOG6VJditHoglMQ9Sw7egByYz0QiWNaR9k3pgRLuJHqy0B7yWsGwtqxtQXpiPrKG86lrdgCKb
zQDttbpBKyLWq25I2HFrUd3AlOJcp+zkrQswyTQIzDTUs1ynhnkZlJZTxVxuPi/DqVKuLtblJwV4
S6aoJC1F1tryN5ZerDzlwgg5YHIodrMD4dicuBCzRIKxeM1FIcWe2JT67S4KWf5+Sr6YRSGJUXlS
2cvbsiaV5aniPyEQTGBNKssTzPYJfbKta1JanqQCn9DnFtC3+9ry9CttpIE5dcYrzBwlVdeFGJq4
fa8LaaZvM32/hum7Uy+l1NIQAxb2ujSkEn3bXBpiPOZlm6aaDffylfFLKnwkVvjsbmV8TeqakGtZ
n5XxEK8Vr4w3SfhL0dL0q9vfQMoWYU4tRpG6rqDJmhNG2hJOVqlykFVhma8UyER5ySqo3NcLZKLI
nCrY7xaATC6lyMCfZXVBqzTA2VWN8aplWuqBnarLtCRCcw2QaQhkorTcehSYgyo7KilNdrnKsf6G
lbfAkYOLKHdamuwYV98ieYrvszQ5oUDssDS5pHDE9kuTHdPS5ClB2ytNdubKgbUxJZssTXZyS5OX
0qIx+bmN0mTnaWlyYQzCDgnGIJXWVjzfyCvlWtWmNJlRV+k09BI532yDvYaBI9GiaP/ub2C2Ewb+
6H13OG7jzxfRZBz1wzfD7n00Oo3fRVEYgfW5vzkNIbJ34bOpYSwoJR4GUQocw5DbKuSMKMUcIUKI
7DO1wxmZPG2UmHHQvbRswnIYDYnrB7YikWMLTsG/9D1mS8bgqyx8wkNiwS1ZjalJ6Ga9mO74eGhJ
Bs4O+Kgua1mBdws+7fjzocWUFlwKqWA4La8HY4m1noeWcLAXgHCUblm3nz6PuoHXm33G4vbNhVoy
J1PEuB9l2unMSDXdobO6SL3+glTHn6JMsggo9qO/rtJNLxUnmnHOqEKwMJHl6Rs4bryh6gFWxSQ3
3IdJdWH6luwhJqOINyV1Ptk9EbrQyJ50XI74kABvzJFAB3hmgfLAECZFml108PoJGJrgQnDodjDo
Td9sUlxotg9wNm5rbfgPgWXkYfsgX2gCgoh925da2jqIpad1DKFTxZZLK+dw1bYbI1DGk0Qlz0Yf
X5TdOBfUAGtUjxple1C2lpWJ4mhWEGBKelgm/sNdPzj61ofPT2jB9+rTILR+vWl3Ol4Mb07twcnD
lB7E1reZ6Wj/emMN/P/gB82D7+TEiWlMCVFxQMl335WyTFgql6p1B8am5w8m6OjNRB9XF2ftM3Af
sH6wnerGnBcM/KV2B1gYgYOe1sofZkb60MhKtx7slMl4Vp0HJcoA5uZBTikAOgmV7EjZLKehDXnM
cpalbK297CtbuHX3sn8kvm572S+jbMd72S+OcMl0jeHg5qZrMlPS/unq7GH52Coin5ghwWZWIJ3B
0e/xiaMb/HNmUVoaxGQm6qUFr+zMXGHZaVQFXyLRstB4QaQDLl0vAtuETnULLG9y6wv6SvGXmPue
jPvh9YTy60kgfryeaKXgF7meRBH8gP/c4Cc8dT3x3OsJgRMenhQfrich/EH4z9/Yl3CQwZU+3nkM
Z51/X0/84HoMv8MhmflHHT3782AzsuOOKApFOgHmgH8YDFFKm8DbTL6Er248LIMePUDqENQAoXPB
WSuGoCwJpjAyk5RlgRlENX92b4+Eng3QSkpJohO0hpSeRmx6IWLT5SI2LhQRae0rbescnC2PweJP
XQF/uI3f6hmEX7uUzcXJg2dwVuMTrfHAAmcAJtQfBzM6gZ7SYGNq0e0D4YnzlSjGg0akb3nUFfwd
DPr3R/3BeAheFP5OlQb/GkXRn6AvBx/Kq4mbI0ZdrCU3cAEmzuFnyam0YZlZCwKzZqVlzYpqOrky
Ia0xqVxEnypLa/NTSnKSbqoBUyoPul6diAW2ZIvwSun055t+Trl2arI5CmNgGNM9D3XeKJp5/6pF
Ed9vkCzwclzCuHYYHl4LyZqTaYNk7R1cKDNua8V5kpIAxBbanu9qkGYM5Pixa0seC+BOu3FIt4Bk
ZXQ3SFYdlK3WSFamKV8okvVkHuwJyZqzI6XbeJnZkGpI1v4s3AaQrIz4GiJZhZTtHslS6f7K1Vpy
mQ1uLpL1pDq6sG9YQiR39Dpq6DKpAt/3bN/zfVu4ng8mRjm2I3zHozICJ91db91eUYfBR+q3u25v
xfvdL2XdXlpDl7MuJL/hyh4W/iR1Xk8IdJLqnP0v/EmjQON1Sbtf95PUW+SsS8pf2LWHdT9JTaqR
AuasAVll5dyqboChhdvzsr1m9jaz92uYvTv1UUqt2jNgYa+r9irRt81Ve8ZjXtJ5Nhzu9dPAquUi
6Pw0DUzJRtLA3Y/9QaJ94d1tr4vY6DTza32bvbS98HKEDDpp9NzJngyeNqgujMgMXqFdxTQn322G
3yZ1uyAlkVejtMfULVhnIRiQmCyQYTnFtmVyt8hgbga/yd2W1ZM8MTa52wJpKVWn3C0hykEzinNK
EV5p1SnwhWvomu3zDFedpvLS9VylW89Vp4nIWKUdGje/6pS5kkpd/BUy1ABGqizT3ML2eZILJ+t1
WH0Fi25R3Oq7yftj3p9q5TCNNY5r5f3nZNrk/feeii05bmts0MNBnkRA2BdSYMqn0nZD4dqEx5EP
Y0siXm4bwVJzuMn710HZap33R03JK3z9MvL+GXd7z/tndFTcp8/MhlTL+89RtlMLt4G8f0Z8DfP+
hZTtPu8/N8JlNxwzG9z1oUsN4XSeGao9dMkggIoV1yWhS+Q3r1C+gS4XpOTWDrpUjAhKE5hFkTWX
nRQy2ECXm9CTBrrMk5bbItSpGXTpcpconFPSzZtTq3eASvhibrNfltF+WYm0OK3HflmMu4wboVZF
TWvYISUtistvGtQKUSumqSMVNUGtliCb8zJtUKu9Awklx616TOd7vuMQ4dmhki4IQse2D0K3Qw/z
LTEF/6fyapUVc5g1qFU9lG3vqNUqDc+r/HhGqFWZebBF1MpAytVQK0MbUoxaGVO2UwtniFoZEL8n
1KoSZdtBrYxHuCRqZTi4FbfJT4jTzu5WuNdks0bkWut6rHDHpiGC8HSzRmVYP1bIGIWA+FmCjyAH
CTFvybrJQn4b8HFBSiJvd469go/g80mWACXaydvaoQT4mDBIG/BxTfCxUIwN+FggrbrVTTqaZ7XI
kuXVTT54LK4qnkysRTGh0+BPiD9J4XLwFVbvloISLVSVOZE28NPeEQHzYXPWamrmKQI+bgCurgR2
hJLK1oQHdhQz5snAk4SwaujTignskAZ8qoeq7Rl8MtHv54s9lZoFW4OeVsqYVkWeDO1HEfJkTthO
jZsR8GRC+15wp2qEbQN2Mh/ekqiT4ciuXSsFEz/fha47XBEFhEpfhGXhCuQ3r16hgSsWpKTXqyjb
QmiFroJONhR1iLMmXAEMurl4TANXlNSTXDE2cEW+tChdp7JsG3AF465MlnlKtWJ31yU5RpHmVxq8
IsErHKWUXrteRqRrZxvAoiZRZMlxq+7UC1/Q2GHg63mU2iLi0nZdxWwfTC2PCAQwbryFepmM7gay
qIOy1bpeBjXlme/uumIe7Gp3VwMpV0MtDG1ItXqZOcp2auE2UC+TEV/DeplCynZfLzM3wiWRC8PB
zUUujHr1JvRptutmxtqwX29KHpf7a2a8jIItNTMuL5xtNzPWZs2MM4LUNpsZ67kGwq4xJZtsZqxz
mxkvpwVHZBvNjPXTZsZLlpxS7IUiv6QOMKUwA9miVD5HfDYkxJXMLYvPIr95ePQXj88azgUqa9Li
WkkB3/Vk5SprS0M4dgljLFfN9wk4ayaUTjZ0Em57XcC5kMEGcN6EnjSAc760uLtONeHG5xSVyiHJ
cnemkpPV8GaVxicN3gxXEyK4y10t1sSb52Ta4M17hwBLjlt1NIaS0GWaRDajPLAFMG/7caRt34+V
4qBxvudsAW9Wizhbgzc3eHOhhud9878MvDnjbu94c0ZHNbzZ0IZUw5vnKNuphdsA3pwRX0O8uZCy
3ePNcyNcEm82HNyl6zOX7TBMcUteVcudYGq5iTXKi4layquum1ijyLiuxybWnEg3bdrdlhU3sUZ2
8HwdNrHWrusu2ZO7VDTtQDSdV75Vdyg2ppp4YAzKQrFF/H7xUGxJKSlWs20F4UuoSFLUR11Q+/WA
S+SvaYiyNnBZJMYGuCyQVm79+f4qZQUQ5Ih0rXy78sJeDY5u0w4hAy6VIByc/dWFsktX2cyJtMEt
9w4llRu26kF9wALFdODgCFJbBMSxYRQDm4RcOFq5KlIVYUuDCdyglnVQtTov7EVFedatEFbNgt10
QjCRcTXI0tB+VFrYO0fYTo3b+gt7M9rrt7C3kLCdL+ydG96ScKXhyC7fTm7JVswUt2JuNq423Lga
pcXrsXE1sEHSTgCainXLqdyWcp9l815OlC9lXBaqKuK3garmpeTQ9aS0jea9EGrRDKKl6zXvTRhs
OmCsjVUVibHBqgqkxddB9rbRAUNSzbPaXL28yq643xkjaa+nBqxCsMoBqVIi1+zdOS/TBq3aO4RQ
ctzWqEGJIEB1gCkWxtIWJKA2TFEGcvV8EgF/kcM337szodtt8Kp6KNveq+xWaXheyuXZAFbl5sF+
endO6ahYZWdmQyr17pynbKcWbv3enVPi69e7s5iynffunB/hslV2ZoNbuQsCEKfdr64LAnDtYm1h
HdbzETD04MqmXRB4TiD+n9GgP7wN2hi43AGJr5ZApIymUEN1a6JlEDPH9W1fBBp3QcQKPe7ZoS9i
PCGYVKX8pTzyi74WGfWVvCZ44QA5Gb0Y3OJeAqOjVB/Kvp9X/Fo9vD/nS/XfuwOTyscDDBGCQYie
FsRBB14wvks8n/fDO3R87iHqBs7gAE/OppsUPJ4PUUfh58FMDR76UD0I2I8RT8Jzoi0TB87rdae3
goMErpQBQGxOIDEgMEVenxDI2iKXQCNXdYMSTBfaPiFPtVUueSambaMDnNqNJ/SRts6lz6RaebPj
m9a0Ging/8pbOV7VmzO0cI/eXDN7m9nbzN4tzd6d+ijLIp4qLOwy7tkMfRuMfqqPeckYyHC4196T
G4nkz3FP7pgiTE59XS57W8hvk72dl5LW6+1cvvntHLiQLMndctI2LVooDnoK+GuSt5tQkyZ5myct
1qKkVh3EgEQuktQtI8vXGYDHWjyZstYrTe4WrgZegHItVncQS0S6TFceZdrkbveeTis5btX9fMeR
gRYwhNqRMQa1rq0ViJQoOEAZkcLzquZuV8/hJndbB2Xbe+7WQMOfc+52xTygO8rdrpJy5T5ihjak
OHdrTNlOLZxh7taA+D3lbitRtp3crfEIl8QtDAd3fdxCgDP9LFufs4DEseJlcQvkd70uWV8DbgFS
4nXrJUYp12mUxRVZc7FFMYOzBmyV2l9mn6TRo+LDQ7LLXtBXir+EGZDA3+mu47c9iHkO6GTcD68n
lF9PAvHj9UQrBb/I9SSK4Af85wY/4anriedeTwic8PCk+HA9CeEPwn/+xr6Egwyu9PHOYzjr/Pt6
4gfXY/gdDsnMP+ro2Z8HpfCdQjE1+E5JbRNfdcs111UyJozYsZIBeJZSw+tjZhPKYz+KVOhxf7n8
JO41lpTnXEAYkPqTD5/TmeMvQDzgUfZBufBL+tJ6d3fjR0N04IfpYSsR4qFF/0Go6wUiAIIkj7Bh
p2t7jo7sOPIDEFYUx9h5oQxV1BV5ZLV/Gwz/jIbvsz4LD1vwvwPfPaUy8QIsE4KmbIBljUHoSesL
OIQPSD+OLNYC1O0766/u+NOh9WLjIU6rATQeY8yXm9AitU4cwCR3HEVcO+IhSIgBEa6Unh1TNwhg
ZjESiFJIx2YYSiE3KeYht3twnePPC6pXGnaLwb3/lDiViN1NEcwEzAw968VUC8vFv3i9ibbtf8QJ
zEeBVAZU+Gm3YY1JTc18x/F57IR+OWzLjKElTgLyU7HyLhnLVCFejMIM8zI0HWUxsu7Nx+wGQxMB
mjF9g4kd2pRq9HC2COY8asY5iM9DW392N44mqBcQMw+8sHcPzufDSesA7rTSM9ZgevTA+jgYJ4D0
wyHrBh+zCWqdjFrmkiJfJijnwozuwoFl99MDIzyCTAGjeMugH3c/WgdWGN13E2AAtTiGECMKO/gd
GUEo8Mc//y+p9r1JlO7VPz+kQWjUGd2NbsHvhEuz24+o9dewO446gQea2kmmxRGxQHc9H9tN4Wey
Aw8fDDtpZMWtuNuDg/gW6+D64MD7e/ZVEJgQEvncEzH1Aj9gZO6f4H/jPa3kzuHf7W+Sn9YH63+W
9bE38L0e8GOhkwWTsjP+fBsBgcBgFyLocXcUdVJi8QIk3frL6447YPUej9zBRSAtLIIGPuCxCHTe
3eJjh3Cw2+/cdPtAvCQPB0LvMwoNrz4ACfcHn8BHgfejuO/AXowsPzk8uovj7gT+HKHV8NCJP/gb
70iFixHnbRQGvTv0IS17YN3ddcMWjknr/mMyNi1vPB62sC1WaxR97CTNjTq3USsbjNbYg5ea5ZUe
PUujDZtRU3Fvsia1hqk1xaRkmrP1mg+gTN0mtVYfT7TkuFV3PzzqExYzcD9UpGwBPNqedkNbQXAX
qyAOeBhuvvnAlO4mtVYHZdt7am2Vhue10HlGqbUV82BXqTUDKVdLrRnakErNB+Yp26mFW7/5wJT4
+jUfKKZs580H5ke4ZGrNcHA3kVpjzjpbsmy+9JO4msokg5K/H65JP0MmW1RRVP6Mf05n+XkE/7PQ
/k1iL7HH6CNqY+7nmpPEDg9vcZEChB/9MX6uEY2dWZl5sXByxtQfp7bd8qzHJ0AQCF8hhFCtt/dn
2arSy9+SgArMZym6HpaKcu4UDT0+rdftYy6pZIJxBo1P0n4EcXhGhK7YH3XZutgVzGpSmtkKi2IT
3jIlZ23yZFEsQX8lUXPWFsv6IRhwcwN0w2lF51kBHpJVLKA2b958j6S8tID+Q+uPBfLR31pgAA+V
Z+HgQ8vC0BkCO7i3FA+L04JxsWRaZNP2pYUvsJJRsv6fvbPtjeNG8vj7/RQDv4mNM2U+P+TiBS6O
b3eBeBPEyWKBzcJg88EZWNIIGjmX4HLf/cjWyJLs1pjN7lazR0QARxpJM0VWsbp+1f0nH99cAk+e
/+vfwz6f7VuWHz7/5cnZxe+rq7BM/0Q0WSL4x19+1G8H5IJoWtssY5Dsa+2tT6+C69e3Ke299Xb1
y+a4vZ/a0eV72vaJwg+Pjo4Sd97gwda65fmuXSKYChOkhu4idWtOa7tkdoLt6bd8mKANCoZwCoRq
wmzSYJOUCAFFOFSWUoqZnGAXqZ3dtV1SQrDN3i75XIQvetvzz66D+9n3PGmW89oliTkkbxepW5bd
a4YbYRepnfEF7iJ1p2X3v4vULQ/3bJckOnd4u4Q/xZ1Hxt66Dz7gSeT2fuOXK4Tl01XMXisUK8Bj
F5JTbJs8vXou+TGOD26W+Zxm/8kr/DFuZ2VI6Uz3fYz7rvHWx7hvzxIZOEvjP8YtERftU9wMtj8c
Ij+/c3z18eQxwuTBPJ7cc7Yo69qRf769w5kU4fLRbh2ujrr2Dk850BbHE2BzjtV4oGcmx/liqsj5
KvXM5Dhl4t4OGd5/ZrKSEvO7T7BIjQCRc2r2BGcmcynZ7kCOrn1A01BZBkygte172fZVRATLpRzY
9r01p7XtO3snro/f2JCmCJfOWoIscMoFc0I4AcnDoKiJZ+dobr2aou27s7u2fUsItqLbvrtIOdC2
7yfrYKa276080rftm5hD8tq+82W4Edq+O+MLbPveadn9t32jKTyz7Zvo3OFtXxlwuusortI7lx5C
gr3r3bmM462dy4RZKq5zCYmSVJBLyoLDjj1sB9i5f2ptXfaNkyF7QT6s1qV8yuCQfSgmWFOCKYpo
ez+AqH2PJO85ERjHE4Hr+cmJ5yfH2WKkiPOTCeFctMfXKNnl/ON1E5nn5a9hhr/dbM7uTqdhTHwn
RiQCy5ttls3p3757GTW8PZsrugmFw4k+Dn8VitTN2U5zvIrvvN6AwBLv4KoVB6/c6WZ7ZvLMZVDe
3iMhIMtfYqX34ux9LLffb3va/eL7n1p7zyIQfBh8L+Morm3AyzYgpiI+b83pXW3AT6d1z7q7Ma+1
FTh7dybDd/mwzBT0QqNgkrYYhMlloGkUC6OD0jMntYK4Vzuw93quLcESgm6elmDPSF9iW7Dfepi0
Ndh/tvPag4k55aP2YK5195r19rUI+w/gPtuEY1g3Yqsw19s924WJju5sF/Y38SrRxF8J9enVaUJ/
31ysfWzxhewYi9WT9UWgll0/xMW3X/3vo7h5zcn2UaiEHqWUVu0vbts6PHz56Hv9fuvso6erR+tN
W/u3P9fHa93++CYbxF+Kl6r2davjtyHD/BK/zewiJOk42wvb9lnKNe1ZSmJ99H+t4WHYLza2HczL
v3/3+vsX7eunccZ/f3OxbofJmCQ8SqVg/NnOK/Ev8FE7G5dXtPjCH3Gi/vjHqzeXM/tHkidygrnd
SItxchsIv718n2/X3pnfzbFr37BnwX0ZT6+vtpQKdHgRO254tTnTYfndlWv3XZT4/RDhzuY845ZB
1/c2l3l0XbWV09B11VeWAzoZvsuvM6322GFogG0UDOUHwUA7aEM1QsLrlnhP1YR0LStdlxF0i6Dr
JWotB6yHmek6W3OZmFOG0fUg3WV21huRru9ZezmGdfPRda4GM9HRY9C1rHRdOF2L+6Prz+T4JfGg
KpgHST31cxIevJ7XyoOzl+YZvsuvjAyDpiEEAqwNCyZhCcLcK2AsDUmUE+1Fvy2Ke6/nyoMlBN0S
eJAs8RTQAethXh4k2VsWJ+aUQTxIBm1bnJ31xuNBcs9bF49h3Ww8SHK3ME509Ag8GE2sPFh58PM5
Hi2KB6/MLZMHceXBSXgQVx4spjTv6zsy6MQ8qIhkHoJGqTij4d2VRgxgR5DzroFhUifkQVJ5sIyg
WwQPdp0ZcVg8SAriQZLLg4k5ZRgPzpP1RuRBUjQPdlo3Hw+STB5MdPQYPEgqD1YeTMnxdFk8SEvm
QVZ5cBIeZJUHiynNM3w3YOsiCpXmHAJCw+gowxw0SFpAhLSYKscU8hPyYN2grZCgWwQPdu1vclg8
yAviQZ69WVtaThnGg3yWrDciD/KiebDTuvl4MHvztjRHj8GDvPJg5cHEHL8kHixZP0iqfnAaHqz6
wXJK876+G3aCmfKN4FSHgcXJkEwA5X34ykgeppfaMCUT8mDVDxYSdIvgwYPXD5KC9IMkWz+YmFOG
8eA8WW9EHixaP9ht3Xw8mH2GY5qjx+DBqh8snQdlGTyolsWDJesHadUPTsKD1/NaeXD20jzDd0OU
NJ5jry3wmONoEgWN0eH6aTXyhgdXIjodD8ZD5yoPlhB0S+DBGC2HzYO318O8PEgH6AeTcsogHqQD
9YOZWW88HqRF6we7rZuNB2m+fjDJ0SPwIK36wcqDqTl+QTxIcck8SCoPTsKDpPJgMaV5hu/yK6MG
ayEoIUAh5gCFugEN0QwwBcPYFMHcTbifDK36wUKCbhE8ePD6QVqQfpDSXB5MzCnDeHDQmUXZWW9E
HqRF82CndfPxIM3kwURHj8GDtPJg5cGUHL8s/SAtWT9Iq35wGh6s+sFySvO+vhukpBGMGO0ZAY3U
BlAnJdCaI+A549YJLo1qJuRBXnmwjKBbBA8evH6QFqQfpNn6wcScMowH58l6I/Jg0frBbuvm48Fc
/WCio8fgwaofrDyYlOPFsnhQlMyDsvLgJDwoKw8WU5pn+G7ITnthAgQN10ruw4w67oGWzgLYKOIg
RsJiPiEPVv1gIUG3CB48eP0gLUg/SLP1g4k5ZRgPqlmy3og8qIrmwU7r5uNBlb2/aJKjx+BBVXmw
cB5UZfDgsvSDtGT9IEOVB6fgwet5rTw4e2me4bv8ykijRjbIEyAMDgMzQgIJkQUNQdgh6L3BE94f
ZFU/WEjQLYEH2cHrB1lB+kGWrR9MzCmDeHCmrDceD7Ki9YPd1s3GgyxXP5jo6BF4MPyv8mDlwYQc
jxfFg6xk/SCr+sFpeLDqB8spzTN8N+DJKc7CkLwCAgsEKFIaaMMUaLziLFiIIZtwPxlW9YOFBN0i
eJAePA/SgngwWz+YmFOG8eAg/WB21huRB4vWD3ZbNx8P5uoHEx09Bg9W/WDlwdQcvyQeLFk/yHjl
wUl4kFceLKY0z/BdfmXksYYcBTfyYFqYUSmAVMQBhzWCSHBh/YTnD7KqHywk6BbBgwevH2QF6QdZ
tn4wMacM48F5st6IPCiK5sFO6+bjQZHJg4mOHoMHReXByoMpOX5Z+kFWsn6QVf3gNDxY9YPllOZ9
fTdISaMF0U5TCoS1DFBPgklUQAAdpo40VCCEJ+RBVXmwjKBbBA+qg+dBVRAPquznRdNyyjAenCfr
jciDResHu62bjwdz9YOJjh6DB6t+sGweFMHCEniQw0Xx4JW5RfIgr/rBSXiQV/1gOaV5hu/yKyPk
XBhRFNxLqgB1EIOGEQIQw15oRJmGcjoe5FU/WEjQLYEHOT50HowjLIUHoy15PJiYUwbx4LV195r1
xuPBOIByebDbutl48NrbPXkw0dEj8CCv+sHKg6k5fkk8WLJ+kNPKg5PwIK08WExpnuG7/MqIQI8a
qCQgJm7ARpADKnwJGi0tx05iZCa8P8hp5cEygm4RPMgOngdZQTzIcnkwMacM40E2S9YbkQdZ0TzY
ad18PMgyeTDR0WPwIKs8WHkwMcfPxYObUHL3RS42Nw/uXXX1lJe5V913P/715Q/lrzq+rFXH66qr
q66uurrq6qqrq+7AV93siqS9q67qAOuq6xPGddXVVVdX3UNYdTl312dfdfuMU/Xu+iR311W9u17M
jc4M3+XfZ+JIegy9B8bYBtAmWCMdMkB7JZC3wjNEJry7XtW3hQTdIu6uH7z6lhekvuXZ6tvEnDLs
7vo8WW/Eu+tFq28FLOruejQn6+56oqNHuLseTaw8WPTddVQED4plqW+vzC2SB0VV307Cg6Kqb8sp
zfv6bpAOjfoGWoODSb5xgCoXrpXUBF96Lhw3REsyIQ8KXHmwjKBbAg+Kg1ffioLUtyJbfZuYUwbx
4ExZbzweFEWrb7utm48Hc9W3iY4egwer+rbyYFKOJ8viQVIyD1b17TQ8WNW35ZTmGb4bsE9lyJJI
cQOIpeGCSbQHWmgKYgqFnghOJZqQB6v6tpCgWwQPHvzpnaIg9a3IVt8m5pRhPDhIfZud9UbkwaLV
t93WzceDuerbREePwYNVfVt5MDXHL4kHZ1ff7jNOVB6chAdF5cFiSvMM3+VXRkYYIjwhQAQ/Atq4
YJKWMvxjhZbKI0mm5EFRebCMoFsED4qD50FREA+KXB5MzCnDeHCerDciDxZ9eqeQZfGgzOTBREeP
wYOy8mDhPIjL4EG5LB6UJfNg1Q9OwoMSVh4spjTP8F1+ZaQMg9YpCKAkDaDMUaC1g8DGrmrjUPjE
CZ8XjbZXHiwh6JbAgzFaDpsHb6+HeXnwOq/05cHEnDKIB2fKeuPxoOxU6JXCg93WzcaD197uyYOJ
jh6BByWqPFh5MCXHo0Xx4JW5RfKgxJUHJ+FBXHmwmNI8w3cDlDTIwVBScyAIsYAqgoBSAgEnPGsa
ZTGmE94flFU/WEjQLYIHD14/KElBPEiy9YNpOWUYD5JZst6IPEiK5sFO6+bjQZKrH0xz9Bg8SCoP
Vh5MzPFL4sGS9YOy6gen4UFWebCY0jzDdwM65Y0ymigHnGFhMjw1YTI4BZCr8H/mHHN8Qh5klQfL
CLpF8ODBn94pC9IPymz9YGJOGcaD82S9EXmwaP1gt3Xz8WCufjDR0WPwYNUPVh5MyvEznq2Uw4Oz
n2i2z7iqH5yGB6t+sJzSPMN3+ZUR1EZBLjhAvImTQcIFk2IOtIAMhiizVkzJg1U/WEjQLYIHD14/
KAvSD0qZy4OJOWUYD8pZst6IPNip0CuGB8vSD8pc/WCio8fgwaofLJ0HSRk8uCz9oCxZPyirfnAS
HlRVP1hOaZ7hu/zKyHOhdIPDPKCGAdogDCTRBDhlCQ/BxDw30/GgqvrBQoJuCTyoDl4/eHs9zMuD
13mlLw8m5pRBPDhT1huPB1XR+sFu62bjwWtv9+TBREePwIOq6gcrDybl+GXpB1XJ+kFV9YPT8GDV
D5ZTmmf4Lr8yItxDiJkFIlgQrpocAS1oA7A0TXCAsKZxE/Jg1Q8WEnSL4EFy8DxYkH5QZesHE3PK
MB4cpB/Mznoj8mDR+sFu6+bjwVz9YKKjx+DBqh+sPJia45fEgyXrBxWrPDgJD1b9YDmleYbvBuys
0EgnDfYAaWIAFeEDNBRhgj2TyGLFoJ6SB6t+sJCgWwQPHrx+UBWkH1TZ+sHEnDKMB+fJeiPyIC+a
Bzutm48Hee5+MmmOHoMHeeXByoMpOX5Z+kFVsn5QVf3gNDxY9YPllOYZvsuvjLiAhlGvQfC8jAMz
QNtw6QzOpRx6jCCdkgdl5cEygm4RPCgPngdlQTyYrR9MzCnDeHCQfjA7643Ig0XrB7utm48Hc/WD
iY4egwerfrB0HqRl8OCy9INKlcuDDMLKgxPw4I15rTw4e2me4bv8ykhDJBsrw3C04HEDtgY0lktg
oKeYaMydtJPxYGt75cESgm4BPNhGy0Hz4EfrYVYevJFX+vJgYk4ZwoNzZb3ReLAdQLE8yCAqiQdb
c7J4MNHRw3mwNbHyYOXBz+f4RekHP5hbJg9W/eA0PFj1g+WU5hm+y6+MDNEKB1cCLXRwoxPBOM54
mNEGYqmtNhBOyIOk8mAZQbcIHjx0/eBH62FmHszWDybmlGE8OEg/mJ31RuTBkvWDd1g3Hw/m6gcT
HT0GD1b9YOXB1By/JB6kJfNg1Q9Ow4NVP1hOaZ7huwE7KzTYY2cUQEqiMDDiw2QYDKDXikBsmGmm
21+0tb3yYAlBtwgePHT94EfrYWYezNYPJuaUYTw4T9YbkQdL1g/eYd18PJirH0x09Bg8WPWDlQeT
cvyi9IMfzC2TB6t+cBoerPrBckrzvr4bpKRhwkhnNAIOo3DBRNaABlEMeLjweqka4jWekAerfrCQ
oFsEDx66fvCj9TAzD2brBxNzyjAenCfrjciDJesH77BuPh7M1Q8mOnoMHqz6wdJ5kE3Fgz84bS7a
gjZcUOy+DK9ipR2z9sZsjq2LyXsTVo3AX375/e7Vb3avHv2XMe4sfPG3k7PjEDa/hIR/7N7o9tUn
q8ufxggym9PT8BdtVR0zFcLiCIb/0JeUEqw+sW5PFCuEu8yTXeZ98+HHNy6wP22jRXGhr8+dfbON
BQVCfSao2wKE5H4TdrMT3/7J6vJ1d3l9jH+x+u3keJzJadbtsg3vF2IoVh9x+45/nhx/asp2Y965
4Kj40Ztf3fnql4uLCOs701prHn/xwZ5QyLUWPfnT15cf8c9X3/7w/YvPztWVJSLedrbbo8s/e+3O
f3W35qVNvSHtXOjLHLd7ZXX5C+ctkH08OT+2kwJQWJThQ/daQq4tafsGn1ryJjgjXIu2V7ZsQxnz
eRviRfY8zFcPW7i4tqU9OGNEW9r2xZ9CDbbe2LV5FjdbusMSAp8iyWu7Iq1dcWtG71qDt6e0dipm
h8Z+bssv16XlUmvigLZUAcriLhBQe2CwYlzppvFI92pSJC9gUfsTZYTaPP2J9PgWC2xN5K2CsbsS
veY4ryGRmD8+akhkGHavyW1fL6KX7ffZhhho2IgdiAz39mw+JHq2s/mwSyJHf3396tUmpIhIFnuM
ZIh2JCCKbkzXq/Dq1/Edtyfxyw/v+vjNZcm7S05PVuEj39z4jRVYrd+ebtpAs+/PjmMjw4Us2/72
6qvdhx599OEx3725XPtvdu8crsYhSoMrrnMthAR7aeCfH9J4rX1Y4zVcZoy369mvZYxXuRz/Lne8
jX1Y/vXkYY23gQ8rnpV6WP51/GH5t6EPxr8EQppZXy13vA49LP/m1Vd8seP15mGNV6Cc9bvc8ebl
5yWO11mILbbywYzXxGYlwQ8lnlv/cgUfin9DfUWVy7v+Lni8/qHEcxgv4sbShzVeKXLiueuG0DLG
m9e/Wu548/o5yx2v8Q9rvHn15HLHC7P6V8sdL82qn5c7Xv7A8hV+YOuXZOXnLm3MEsZLkcZjjZcI
emO86605cr+ZFyftgzbmF2fe/ffmPI7syerZ++35s2Z9+ize2N+6ixUA5uw9OF5vg2kAk+tfsHa1
9s8zn4w/0dvwu1v7zG5O3pzYZ5v3F8HQ8IbH+u1zuz535uKpf3983BxvzLtVs33OEF6ZzfvTi+dy
tX23PntO5eqx+R/bPpHypPcsETVoll7/9OLFy9evv1x95c7P/7x6vvpC/gdcBas353Yb3Pnz6c3v
w+B+PqVQ8VXz+4Xbrh7TI7R69/WTMKCztbNPV/AIQsYZ43S1fboSmIafPtv+fPrFf66+Ojfx/WHn
CO9+NOmOAcanDT8f9vHppMtxfhzx2502I77j6mKzCrEcVsG/vrgRFPG5ts1ocbE+DYa3j8q1kfEh
JC4/5TpY4vdmc/rr89PNxfn7UxO/v4ya+NXWuXchYL749zhxIu8Ok5PwCy9/c/Hbnmtp5DlbfTRh
q5uztbo5VVerazdJQ1YVRwNma4I1JSQmCMU1xTrX1O7ReUzF3nGpKGFqZ/H1N+HagfFdA3rrLn4I
b/iNO9a/Z6RSG7zfINIopw1AHjtAtXCgMVYBqjzzkjFDrHx24i50DJNb+bKNoPgWp8Hn0ant7Fzm
THTTq6mjxlmj/tSN6CM3or5uREgphaMbyRFdvfrEjR8eWENxi8E7h4SeQjLoMI1GYSOMUHHTFAMo
awRQ8aAFr7UkWFgOeZP3tHVr+F3ZfGd31uPWwTeXT/nGJ4Qfb+3ucevEJ6P7Pp69Pnm7+wOiKbPY
NgBiGMPYa6AgNwCjpkFUIqfM7nnu3R9Io8PbEg081lGPaCCQSEvgSfgwGb6lOP7B5izKubbPLyP5
xvyhzzgeDXF8WIaiEZ4CDnGYLkgR0NRIYBrEuXVcUENyHY/2Ox4tzPGCUKVC+QoaHvcZIg0CDUIC
cI68p4iHMsfecjzTRkrESEh40fGM4DC3vgHYSMu8FxZ7ebfj9z6iOtjvDXHEem+BhQ0GVCgXRm8N
IAYZS7EjQvc7MyzpwfIlut0aIQT14bIWytiwfMO1qwlzBBRW3AjSMNrAW26nBBpHEQK2iY8AIxqm
FlECuHLGBpCXhKl9651+xu90iN9F4z1BYeRcIBNFAhI0QjJAoEchxwukKc1d73S/4+nCHO9xyIZa
NAA1jANqwz+NgQpI6wzDBjcNN7ccj71nFFkBGul0mFtHQQNDpDiLmacQSSLgJ47vVYj+P3tX1tzI
kZzf91fA88KVjeLUfTA8jhiRtJYW54jhHNbKMqK6DgorkKAJgqvZ1fx3ZzWI4dUEqrtxUaIiJkR0
N9CZVVn5ZWZVZlIwRKsOBT85wXdGSW+cE8wVVUomS09LmekEP8gh74qU5fokB3NGSdJ2o7SMYIjA
pRQonhsKeZA/CfxVtc94koI7oyQ2TwooNqz0+5Su0gbTMiIphDAGGp/PMgpk15hW3p+0AuOisEhp
QC9wCSwqfFBIYewKYUkEyKtlFFSR/5BpcEV9I9MAXjhMnIz+fBtUW72/RrLj1/dXJDr+EyzBCF6B
AZsEJBAMAlNgZHTBkOeFczgaQxjdStnlbuhTmi7udrasuxiXabPvz8cpa/YSFg5wBhdYeXdSZeP6
vk+xCfi4lcQKU6m1SE7F1gDW3G4S43TPbKsy+9cO+tOvfummPFxVeKzTiGjlEZcefFlKJXKeWupE
sNKGfAJxBoGMSanMPQLVtqwkMMvIW+AISgE+nrxHHiXbupK+nJjWQmd4EjC6R6Asi4ncp8/hAL6n
A1+EJ+cFTH1Y2RYjT5QKmhREE7/YCaZGa3p/BKsk8Et9NUeapgNnqrjrdOCn5fu0fJ+W77KW70qt
lFlJ801YWGXu/GLoW2AKffM5r5lJnzndlZn0030ggWcQp7o0uQCxPwhpG4gotqTNr3tO08npxfP0
5zkQCy/cntT72HbDk50eGLK98OsZsPM8Z/VPXaoF7ZvN2muA8TKbOV4n/eNz60DJPc+B7MUO2RwR
Y+lQUu0hu+9x4n8jtzzOm59Lj5NxPHU40593/E1MNaOm3C8m27RiozFTAhgxq2KHX7PD77ODhTB6
wo6qYKdWhEB3wW54iqPMHyXdLtq0jKiqUZSWR4v07GMQdAZnpstEk1W6fMXWHw1Hz3Os/AXDwJzR
UqtSAox9VQLw510lwAwvQ2iai4qpx8RYx106zcDAtgdYQFbpgGIoHKy0EGM6TziDTZECQYNUaZAa
/BCPrvp8T6fz6eW71wevv9vpwC+kSen0R53z8elpMlKK8UXH90e2GAS/3QGDLNW7/PpgEeLwPHTC
KdxPT/cv/mXRnEl6xRmXN86+HH58lRYtmKVD6weXo2+AJCC6/Aim8gJey6pe++YsnJd1kl+NL8Kv
dyj4ehMcksuTK2I6w+nVrc6VEe3Leo1fr3dO0m8tgmSeKQP5ymA09sMOOp1cGKUribPBpfvZnh6H
9L3haewfgxBNuryA2Z1cmAiyFXwvVfQbgRj8+Ox/yzV9UsYmnz/7aXJiN/RG49FZOIUJ6119Hdb4
38/7F6HnLOjjssxxeIGnAjipl9crC0j3JjqBgXs0gIvpLZ2t/9nasr/dfBWTsPZCwSwYW9YVjuJb
/3H2W/pOt/zm+W/b/1p+7PzU+dLpHA+GhR0AP52EUCDcvYvPZwEIBAb7QyCyPwq9CbHpgUR65++2
f9GDFXF9ZQwPXS0W4AN+NpUDHJ+lnz2Hi/1TMAhPgXiBv17w9nMatPT0FowwjCY4ST9nFdjLqkxd
qUVneUYgWnq6CMv2sQ+vhv5p6XbDrF0e318R13dvrorj4UXlgmhAm1g0bTNXbDaBU1UiSObgVamT
JQ2emNJGMwcvn7bFDN6UQKbwfD1cNbP39XDrQZMrUrTAzu9Bx9Z4ujC3nv60t9v7tCfAfnr57Q9/
RZiQ/3yJ4Qr69OrlD2/xfym290Fuvg4/Hf4MggXvT9M5Bu9l1CnKy6NxjP1f4c9RCvvZdLh/67f0
jcnkJT/oLHg3GCdnqoOGnfG477tpzrv24uK8mwrYdwEfQreszXpRVrSf/l3OXnn36s8LezzqXh6D
8W0nD179Xf7A4PLqqbMbf6Q3ZRbfbuaqcQyuWlXu9ZNDe3uUFN64Y0JMKsPMJLBBZsVpZpxw5+mE
u/mjnetPXKf7m3Guv0zQmJzr1y3jU5x2FXk67ZUxSmzj4lOMGazMw2HXWhxy4LCqZNmTHNwZJdFu
lJYRp+QCT1IgDcUVwarMZjnAnElsrLKbELtP3YyxNyk2uL5uQomCtNm5qm5CNQcnkbbkbkIM53YT
mhK0tG5CDN/q4PNwOtBdShbYTYjhym5CM2gRk1SFJXQTKmmZdBOqpdEkaLSn1PeMUVIbZ84LhZko
t10V0TP2p2btInPVZRRv5P7URh5USOPFNnM/b1MPKqQhS7tdG3FQQSqt2x1UKNlpclRlCQcVKFbq
6tyFbOsA6K5KMPkEBHNGSeOqSkBrBgJ4YoIDrOE5BW66LNnRG6jXNvCcAowWT3tXm3BOQbI068n5
I21T/wQG6W5n5vwBdEAaJbZxOoAbJibBIM2qggA5UU6Ropx/uNhu4jpFODYjtku4kWIS262C9Hsn
pGdoKUG7JDHW/FQ8c0ZIJwkCJVsgDtYGMo5RpH1UwdnU/sK0y92jD/rKU+qXm7tX5/2POnevNBXv
pV7wB3JX1pD8U+LY/eyaBwjckOQfprbxpiT/JMVxj0DwczYl+af0vbIksCIRJFvN1c3dy1Rx687d
e1q+T8v3j7B8V2ql1Mrdy2Bhrbl7jehbZu5e9pzXzN3LnO7WXXCBSM2fImIZoyTbbSAtZ2tEMj5/
a2TWwVHBuobSzczSyKb/a1IEmXlauM5h3NanhKeEnSUAGqWTj3t9cJp2KLshMW/v3Pzz1bH6bzrp
BR0YljT3H797b49fXYUI3n365sWPW7uHL4+OXuzB5wRMe/tHu+8O3r4/ePP6BczaqI9LK+rgzRu4
9mr/zYf3R/u7L0hpBRzuvzzaf7f//t3B/tEL9vVKei49JCcPvdn9/u2bw4PdH15MP77bf73/6eXh
wev3++8+vjxMz4rJve++Pfz+6OCv+9OCym//8sPdK2/eHPbKEm+ZJeTGW28/4hdnlztzT/F209HT
HTm4+P6Xf6D4fz8fozfv8Wv0PQ8a6cFbi/5B2S7avTzjlHXPJhK4g+GvUjPsEMk06Z7Ys2GMoLp2
Su7fvTncf/GqVDfp49FeSXpmtbyt9z+83X9xcLR7dJA+fdx/d5Smpbz18bvylwZ27+gXhsYnv0r0
3yrZN5/G+0ibvXPE/YeAXl0O4snP6Qu9o7+87O1+f/Th1Qt4a1BSU2JwEZ1y2hqmhNJgIyluqfJK
G+ko2frplng+5Pcn6UwZm7C2Ob8RCbr6/14Znp4smuMkoedT4HydTnafWQDotF8/udY5nV7MOpXb
65/AK14fdewgEfq5M33H9RGDbNoFXjHtl2UhxdbEX8fh0uGVlXIACu+lu+hflhptPiMzFdyqcrsu
jwHTfwc5BxueE1D3TH1WjeokKFw0jDHeKl1aFWfcugBvLJ1xKz1JwyUBs0hMYAx+1J4DuFbd/pJZ
GbykXjYNvGRWBb8ReFktPyutdD7LE61H/Cp90LaULdD7bDLDNf3OzMmt9DtrkKdaCaAA/9fFAllN
AuJEalRYTlEBSspxqb02avGl9qd0P6oKzGAues80QKwyqUi5DMgy51Kg1BbYCiKwuVWBmWmtA1YR
WQk/zilmyOIoEbEhSCGwjfZ+BeaG47c8GNBgmUhjJNeVavPG7VqLSjWGgTyJfRAGls3PSlfhgmBA
bSwMVFK2HhhQTWEgb3LbwoBuI4BRGSbhd1FUCgONGqAqBUtdYIJainVBGxfin6PG9CODgYAZE9oz
JGDMEJeKAwIoAe/j0jtpKb/TeCOnYkMLGNArgQFBGeeGsEm5yLta88bdWktKNwWBTHl9CASWzM1K
V+CCIEBvLARUUrYeCDANISBzcttCgGnXi0UqXRQCBe6BRqwZ0oZzJILjQhEpRVwWBJjFQkDWXnyr
JjwRYNtJiZim4Al4mrAdMEZjrK3gAn7/dhMeCk/TUEikhACKnPAw//BCLWBswRkg3vAWEGBWAgGM
UkKp4KpSad6422xJ1YWATHl9CAKWzM1KV+CCIMBsLARUUva4ICBzcltCgGjVecNaLnDqDVgEplPA
CvSakQ4pZiMjUSgR653ezVZh4rH1XbTeMwovQYoAO1xYj2wkHDFSGFfggK273Y5LERYpeAsoAA0A
RUoiLcEicJwJyYpYFJo2hwDRtGdIPQighCuu2QOxkxt36yypa5mtCwGZ8voQBCyZm5WuwMVAQCJ+
MyGgmrK1QMD1DNeEgMzJbQsB7TqwOrB5mAVNaUClcY3BqtXBgH1LC6oLEnBs2Ilzrgp7bK04MbZg
08cIj2mZmtVGZGD4ANGpZs5qrJy4BQFg5BNwDhSSGF4DfglDMKYEyMLwwcngzczWu3XGb3n7AVgx
BR4s5ZXh8+u7tZZU4046mfL64G7AcrlZ6QpcEASsuKVIW8rWAwFN24hkTm5bCGh3Op6qgjCHERi1
SXEGjwxxDkXvPebUEx0aN+Geo8KaZu+trRszUTEAeCeNDl/QiiPLdIGw94JH0B3e0lsQYFS0nFOC
eAEOIC/gWwXn4BSCCyAJIYLxNhDQNPuw7skgEBEmUhOP6pM017drLarmKVl5EvvwyaAl87PSVbgg
GFhxdkpbytYDA6xpRkre5LaFAdZqQ4pTyQIGbPLMIe4MQFWhJGIugvuiVMo/WxIMsEcGA9gp5zAv
UIE5DJVxChXOUWQwJYZarQIp7uwHFCFEQIDAHGCBxApZRVI2nnPRFKZwtk0wiK0EBgDHDNECV9vO
N+42W1K1t4Tz5PUhEFgyNytdgQuCALaxEFBJ2eOCgMzJbQsBvI0AaqsEd14jjKVPQe4AKtFa8FaM
FdhqKUK9Ttz5Kow/MgiITgdZmIiEARzgggCig4pHRMcI2t4Q+M4tCIhRAdSzgAIlFOY/KGSMdems
KLiGlgpWdrZtCgH8UXsCvCkIZErsqj2B9azCBcEA31gYqKRsPTAgGsJA5uS2hQHRyg4x1HDjHWLe
gzJUAdSoV2n/OoQC/gnqqpNU2sOAeGQwoNP+b5TwBZxSPhwXCKx/jmignEammYq3TwYRLTVzniEW
FHgC3gtUGCxQtJxSEqwySrWAgdWkigkwV5ikDFfazjfuNltStT2BPHl98HDocrlZ6QpcEASIjYWA
SsrWAwFN08QyJ7ctBLTKU7SkUIFxhXDBDOJBMNBwOCBsCilsAUarskuCAPnIIAA7rDwvwAmwMZ05
pRgVIRBkA1eScRbNHQjw2koZU/TPaviWpgbexTwqKI4GB4ONdC0gQD7ew6Gica5wpryu9HDomlbg
giBgYzOFqylbDwQ0TRHLnNy2ENAqR5G5wkePLRKJUB4IRzaCSoSfpdw5p7Relhfw6DKFo1CYmIiU
T1u9VAekuafIBUcFwdzoMgHhGgKkoJyztBkc0hcEEcg4xxCRhjJuwUcUbSBgRZnCSzlL0zhPOFNe
V3syaD0rcEEQsLFZwtWUrQcCdNMt4bzJbQsBrXIUHSFacMOQVEyAXks5zd57RGWqayAMddgtCQIW
nCW89BQxwQn3TFrkXSoW4SxBhYocOayV8/Dryt7OEvYRKxIDPEuYhDeA9tfC4lSzUtlCS2uYbQEB
q8kSXlKNHdE4TzhTYldcM2hNq3BBMLCxmcLVlK0HBpqmiWVOblsYaJWnWGCsCyEiAgWagtyCIK0Y
QQZ84sIFQ4LWS4KBBWcKL90TIIbBcESGRIGL5AlIZEnASHrHuVFKOWJvewKUBkGBIl9QjDh3CllG
DQoOK2e1jaBwZsHAw60k74/fEmEAC20ImMiyWm1e3/5yh/g8qa2dJWCVKuFXYspTqgZBljuNXEGk
9EEq7mYfEF0yP43yBPJ4agwDc4iX60sVa0TZcmAgg5RGMJA5ufNgIJu8JgKIKbGaMoyYiuCxFEYg
w31IG54RC8EEoFdTGJitxuRjyxbG2gYXhEJMYHAyIgXjPsGAIg4HTZynit3OFo44EFfAE1EliJUq
HScyCBfEU45pMCI0hwG5mmzhZiGUbJmtCwKZ8rrYgNA8blqlijVegQuCgPWlijWibD0Q0DRVLHNy
Z0OAkbMVwWpyRkFhScGI5pVe9I27X25TnjeutQtIssB8jB55XKQ6nunkrUjb7w4UMaeBKTsrZXS5
zDTyx/IYaqgGcmhfixaYS9jKEoVyKGmkAzJndp4ZOLuEvtzsrME5ZfOvx7Z+AakYGQEDToIZluoy
alQoLRDDkWBZKGL5rAJSS+enWQmpLJ4aWwQZxK/JImhE2XIsguwZrl1CKmtyM2KDM7RB4w6mG5E5
0LytY2ZQc8WZA+363DUO1C5mp2DVHe7aUraWnYLGXe0yJ3e+fzDtOlZNnzZl363ycXgk7bFc9s8v
tqfXtj+e7N1UEL2es4NBr5faq40HvnM6vEg9zDr/7gZ2NOps3f7y3nn/MnwC+To/see/XPWO2/qP
zvC08+N4K5hUESMIYDFVyxDcIs24QVIEGWK0RNG49dMd1ft7YofMYyfpuMfDztzZSTpuXewYUUQm
NQdbOxVodOkAhlUYOcFDKtptjS9usjN/6TwmbnJkTT4eduZMDu9ijNe4cmLQXDqVkn4U4pxZZESM
iMpoYhCMGFXUWTmPjp3ZspbYoetjRxXMKSMYikXKzAYJSzV6CBIqMhYKSwqBa8raGhVBFjepS6QR
wSobUsIa0A+PeAQWkUdOR2wkN4R6Xlcof598z4r4Jq7Vzs7InpylRqxgtzNyxfX02vbHV9+OB78c
pY932J4+0iGq48vGhaM/YWKs48nEEywgLoVBVumAYiiARA5IS/RMgighm9mTdgGc0SvOBLlhO9/v
Vts/LfuNg6wMLu93rL2+e7Nr7TEIX1XD2kVSTWf22G1C9cxeu3mkPxQcmFLeJDhw1Yt3EiCoCA6k
DcwFjWzDzUEVCU9NWsC0SPUbFEdaA5BpWihQJVH54nbC0MIpbrQBmEd1PQd/MUyt0vFfFsULDAgs
gMSmlWYyhaQyULBIspsIOJHEgytpEaPeIm4ERkZ5iQodGY6eW2br5Z22V4GPrTjZavpVLXhclxZ3
lgxryhgllSf5btxd7JKtC0eZcv9QNHq1XLaqbtN4hS8ZwlZc9WZZFG8WhPGGEJYpJMuCMCA7mesE
p6270pdLH8FvuwSnMH6eqJ/9Xy/C6QhU4jedy5MDn+o6OE81BVVOsE2aXYJjCXpYekaxlFRxnjbv
P5a/kegN01/ogOOUFDL8aidHG3en8wR2f1rknesV3u248Xla/Hdv/GnQLxIr+5dw83A4PJszBmoy
BlxIdnMMhqeHk9857MfgPrtBKH+w5hiE9J3kQALHvuNTN/UBzOTwzAJbdUztWQBzxYHAmt+aRUCd
79Lg7Z6Nk2Yaj2oSv/v2w9Qj3um44SnI3nghYgf3e71kyPR66RG5s/O30fD0/Mxtvx5e9GPflb4d
sBBO+jDiR+HUl2JUDuY/n4F2tCejZ4ANz3I4KR8clQMAfz77cPbsS7fz7DS96XPvon8S4KoQ/9/e
1Ta3bSPh7/0VuHyR3YoSQIBvnrozbXO9dq7udZK7XGfqng4EwFhnSdSRkuNMk/9+u6Rky7Ek802i
dHUymYggCTxYLBa7iwUXE6Iwz6V4b4EGn7Z79AWUgOF/FWNNLz4ggT+8uRjkNX4ohKARGVP5rBh+
cg96FliGa5hpNgAIHEdaEQuUcrW0qRI7MQFrnQYriHrv6+eeT4ntCvFhrZ9VDxcXZJJdrZ8I+yox
Uv8cx6Ncub/DvlJ+kprZq1yQIuxT8tN8HJqExNGdxxF7n54Rumb12iz6+X5EfzzJF7ByKytH6rS6
spYjpb9XUo4Bbhjf9r5/fbFw8G8lpu8Ba6U5aw0W756dcU9scnsPwLxU19/FyQU8fEr68zTph8NJ
P+MzMyOWpaZzazRMZ4RaNr9/QGsyjM77yZW56YNMlpYC6pmkX8Q8748lqGpJqvs6Hg/Guh/PZwAU
KhzJt+d6mIBd243mo1E4itU1CdNzh9m4JTKZnfskvR5Oz4VPTtQ7TXLje6kW/PnWqDn0s//UzkyQ
zwiU+inqtSj64niGsmw6P126luF/gAk/gWHeXZkJQBhP55lowNHK3q7Str/SdqaWLdqemNm7OLlO
iwBYPJuDyPXlMSnCarUB44qwAKyH6Va0cqTmI5nhLY0TN1auyZVWVQAHrQG+0bLcpHW6UEWtSft4
18r/gsLKq+JEp9DHy8nqNcy1y4mggUvC9zPQHU5Ej5Hrb3DLcTo0uktoD40sh5O0S4KeA/f66cN9
q2b6t6oLYUe+wRrTMf68q/VkcCUnemReLbxZMHbQ6OAir2Tx2Anru/yUWORbJAuOHWh2o/cYdXg7
m+jLW8Yvb5X4y+Wt77pwRS9vjYEL+Beo7/HW5a0MLm8p3JB4U/x2eavhB+V//dx6BYU2PBnim9/C
Xe/fl7ehupzBtU7oyh/m+auXnbV02rS0bCST7xYhUwq2T84NS/osbpJ0YRVhjWQWk9c/XwCpfu2s
SHL0kcWNCfPhBIBjlbk4v5PjeSv3Eh6vwVq8OQcTCxY+/L5TJxf1+Cs15hqkfOe38ty2bgn0N0+m
MTyAouVb3AEutQA2TDPyCcHIKrXIKqmWS+KCSKtLYeh7joNf0hK261iCKt8KhaOsiAtf+6EbCFtu
px93m9ZW2Wf4OX8RMWU5nDNLyJBZkoI2LQNBIxG4IHGeQuU9RMUCsQ5W75+wKJokL8DRnSYxWEjp
T+Z2lqPMDARSBNC9OvZlBETP9pGhCCsgEtjh1o58wQT9irwbzq7OyMmXIagomuROAPJm3BsMZATj
/Il/DCn05UJz7L0Zkzj8Dw5wVqUXsYhR6kaK0a++6pLfOxPz7nXut151anXy6IgfXkJ54T2UzkSO
TfYCrFJ4nbvYFpUU3PboDMdA8sU7BfdWOtlCO5EjeCkzqqBoCqO1bLnIls7H06a4qLKrAbD4Anpk
GeA+S+Cn6oLIgCrg6DAQyhMhp6U245rpUO6adUTzrtlXedQC9kZjsNHSJ5tpRVqSkyUXnpEiLNgl
S4Y7K+TWbX/Ew8APAbuyOJUU+NtlVhg6rgU9047wFXdluW8+FuvQFiWBV/0Q2IMIlKPffm2ENfxF
GJKwvW1hSIkZxVKvC0HK7xQJmmoQ7VNBbMVVmHSuY2JN8oIUS7BT0FF8JZ5Ew7ekQ7S5GWY+RuTi
CCwMowe4jqRgCvz64l99uA/qDDJd/8VvZPh2EidmkM7TKeid8Oji9XNG3iXDmRkoCZyaOcXNOV0G
z+XL5AAqj5NBbudzEg1HUIitkM5lpyM/rDbFXeBZE3IJRJUqVDZ98EfwD/hON3sz+dD7PLskv5GP
hLwdxSGYfL8TgkoWTMrB7P3UAEDo4DAGkMPUDHKw+ABCJ+/kcDYAqXdfMoeHFoF+0A+oNpTqej7F
ahN0Mk0G4+EEwDv0rkDL90g0fLoDFJ7EV6CjQPtI7jnIi5SEWXE6j6LhLfxMUWpIVOI7H/CNnLjo
/5garUZz1CGJFZP5fKi7OCbdm7fZ2HTlbJZ0ccOrm5q3gywscTA13cVgdGcSGi0y81c1y7J6uF9H
D2/cpqXcg78BGrWw5q0xanOV0bIxu/nGbnldjusssKZ5/RJWXY9v6s5bUI6hvpdmJN834VcbT2Z9
/JmA+gQN9vKVt6fi8dkgns4G5nYaJ/AINZFkillaOKD+hx6o/1piYj3PMz4Lmc/ujI6MeaGJB363
zKjBiTYBMwTtjIys+Zxkq+ywpJfYFg0M9PLoQdJrPHybSAUCug8SxHUCYHzXsXFvPYT1xg85UDBU
ikZBwLjdLMmeYjG/Cos9njH0C/ZgxqxeZzOGC7qcMPjz4XyhlFPHESybMTCdLuq5gbBbzh/RN110
ngT+vgZd3A+6WDPonLGgsUEXtN6gN78OgG5gCzdzbgqvoHNzkz6+sX/PTruybOL+kZ12Zall16HW
DqaUB5NKZBsGjm9v0a2EvaVfAXBBFRm4e11hmMZp3/NCTX0E7Xsa7FgtwY61XUuBUWorx0hXmoY1
qyeoZfM9rRic360Y8PPRisFs3/c3rxjQuEKZNlv4ZNcKU35GaZfiGokOi1jFI23Qb4EzAb8Q8POi
9OWitPe1UmYKP34YT9EXnW8TDWRWekryuyhBYQZOTO67zdwzzPZ6ALvHzoQAU/wRuvUEX8Dz1sHz
18F7eXf7ZBnCdkr+keabU/+dD9GKRvvsnD2GsI1AaxEw3KDdBmFBHaz+lOTlJncN4Rvkdjxqhjgw
yXDZgvqSqQJcdgCFv4xHj6GkICoNDBQ2Hd+AGXs1m02JXkLL0Jx07vDA4pIhOv3sm7yJXy5+fPXz
t0/SaokEZ8eNTnv5a69NcmMe0CULUoLJgYbycoyghOQPJJnD81PiLKYps4WDp2e2IfHvkWRRI4+R
LPcplljSU/LqaQwks+yN/hTLpt3xezjNxD6UazbYVdhDkcPxdbA2EEBQBOJ9xMOxYMVgh+pY98d6
aHDvhvWKHP2vg7WB4SwCsSLrtYi1NOs9wPr/wHo2sxXjwrbcyBOWCAJlBYEdWqCMUxd3SZR5tEg9
gZU1PJxFIFZlvfawlmc91g7rsV2xXpFvS7Q8nEUgVmU9uzWs5VnPbof17N2FuBrJpPAtLwgxMQBm
imSRZ0WOiQIPFgwRlmW9poezCMRDYb3iWMuzHm+H9Xir0dU1sO49sPpYsJZnPdEO64ldsR61jZGw
RFigo2ByRMktSZ3AEp7rR9RRHpCvDtYGhrMIxKqs1x7WZ9bztfJ0qEPLeBgCF/jcCkMtLa4jKhxD
JaV2SaxOw8NZBGJV1msPa3nWc9phPWdnUs8Tri95aEWc4a6c9K2QadcSLORCR45je2Ut3KaHswjE
qqzntoY1Lc16bjus5+7MuWKksaljLFdiGkrt2pjBXFmcR5QyZRS12x7OIhAPhfWKYy0v9Ro7yVm9
2UZZTykvDFnoW1qg+90Iz5IBt2HWuo6mkTG+59TB2sBwFoFYlfXaw1qe9drZSHvQbKOs57oa1BNY
F5wIc3AzO7AkGGIWc7lPXU+6jhvVwdrAcBaBWJX12sP6zHpFPqZfEmvT+6JFIFZlvfawlme9lvZw
dxY+4BvOhWs8i1N0RLnUtUJqO1YAKwdn3IlC7bU8nEUgVmQ9t+k93OJYS7Oe284erruzPVztRbYd
gg3mey5D97u0QkWNxW3m274KKNduy8NZBOKhsF5xrKUtXLedPVx3Z3u4LmVMaUOtgPqhJbh0LKmZ
sST85MbWPiu7h+s2vYdbBGJV1msPa3mp184e7oNmG2W9MAq0pAGzmIO0kgG1fF9GllBGGsmjiEe8
DtYGhrMIxKqs1x7WNVJvc9gqw5Q6y7DVwG42bDXGI6z3ke7e1tjiAMN3s9MVGOoubHtHBwMwOD9k
PAyMVBaLbAPE9AwsITqwRBA5ke84imu/qfj+J3qN57zL9/pxgD/75HgHK3u8gzHKbBsj/HnPWxPh
/+gDXVsSAkHXWFfwxvzFFdremeeuUBxyXcB7D0Y+KsC4uFYA3Jg3pWbb+49Nrgt47wHKRwX4mR1L
xitXANy0r6Vc0PJRAa7Ijo05/OoSa+8xzAcwuuUCmY8KcDV2FI05AWu2vf+45rqA9x7cfFSAn9mx
7peknwbctNut5pekDxlwRXZszE9dl1h7j38+gNEtFwRdAXDTvsNykdA1Ae+bHXfmwC4UE30Ao1su
MPqoAFdkx8bOJNVse/9x0nUB7z1Y+qgAp8/sWC52+gBGt1wAdQXATR9bKhdFXRPwvtlxZweYCsVT
H8DolguqPirAFdmxsfNMNdvef4x1XcB7D7Q+KsDP7Fgy7roC4KZPEJULvj4qwBXZscUQiofE2nss
9gGMbrmA7KMCXJEdd5dQsVzb+4/Prgt470HaRwW4omXdJjvu7nxekZjtuoD3Hrh9VIArSscWI3p2
x46F4rgrAG46IqFcMPdRAUbpODXJMNZD1Q/cLQHFXpfjoYdUjqcjaPAMCnH3CzPGLct6by6+mY+u
X+NlHt49AIgj/HT18hHCQEvIPo+e3scxb8xskTUrKDvIr7ofZgaYnGQ4Sw+QZAeYZCinF7cPI3kK
ZgTAOHnfW5dapOjoc+cg8v/Yvutznn/Y3y4S9r/tPAN0C0/FVk/FSHVoNI18yxFowHjKtaQnQgso
7vquoyKlo1KpGNfB33QaZoG+YuLFaYw9SU9gCoG4Sc83JK0v0z5+L7Vs+13yyszmyQRK8mzaZ+T3
eaeI8OvAkx0Va0weS7ukI9VsniVz/Xsyx1yuNybB7KJQwLO7eYKD+/saWRQuO/ezBNPFjGQ6y3J2
L25lKWlxpVy8+bGLyWEL5N0ojo8WwZcltHgE0N8AsFAGzQYJ6DoiCB7Tz7F7fC2+IieaGh3g/LjQ
I4Buz1mLr8hq0uwA54KtEAd+rCHlGAvYVjGXy6KyEi4CgZZeGf08e59n7/Ps3dns3auOcqeV3Csq
y2leqQuwwsfzRJkF5rOzQKyoka8e3u397d0k0yESMzIyNV+PRqckK+vdlyxT4qTk949kWT1eNITP
p0/jU3KizGgF3l3BKroaYx4EfPuYa6MSE5UcbngDLH4Zxnluoe/AuDL3lo3YCs6zacG0z4/1f0KW
bguyyFRMhilJ5pMJwgjns2X6Zd0DzTDFVEJ3D4YmihNDzATu49PD2Z82pHQrgH9PR7eLiPKm7NEn
eo0nYA/i6DZFDYBnSVzF2uxsdRKT531F2/wQObR+z/Br+3mCeHdlBH98c3H2MCs8ZkzPL0Fc76rZ
ennpF4Jcr01O/z/qONWikogFAA==
--=_0dfc307e5d88262f38c3a472ce71071c--
2:43 p.m.
This is a multi-part message in MIME format.
--------------040408000201060702050407
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
Hi,
If you have set up VM disks as Thin Provisioned, the VM has to pause
when the disk image needs to expand. You won't see this on VMs with
preallocated storage.
It's not the SAN that's running out of space, it's the VM image needing
to be expanded incrementally each time.
Cheers
Alex
On 13/04/16 12:04, nicolas(a)devels.es wrote:
Hi Fred,
This is an iSCSI storage. I'm attaching the VDSM logs from the host
where this machine has been running. Should you need any further info,
don't hesitate to ask.
Thanks.
Regards.
El 2016-04-13 11:54, Fred Rolland escribió:
> Hi,
>
> What kind of storage do you have ? (ISCSI,FC,NFS...)
> Can you provide the vdsm logs from the host where this VM runs ?
>
> Thanks,
>
> Freddy
>
> On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es> wrote:
>
>> Hi,
>>
>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
>> events like these:
>>
>> 2016-04-13 10:52:30,735 INFO
>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
>> 'Up' --> 'Paused'
>> 2016-04-13 10:52:30,815 INFO
>>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
>> has been paused.
>> 2016-04-13 10:52:30,898 ERROR
>>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
>> has been paused due to no Storage space error.
>> 2016-04-13 10:52:52,320 WARN
>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
>> (org.ovirt.thread.pool-8-thread-38) [] domain
>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
>> 'host6.domain.com [2]'
>> 2016-04-13 10:52:55,183 INFO
>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
>> 'Paused' --> 'Up'
>> 2016-04-13 10:52:55,318 INFO
>>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
>> has recovered from paused back to up.
>>
>> The storage domain is far from being full, though (400+ G available
>> right now). Could this be related to this other issue [1]? If not,
>> how could I debug what's going on?
>>
>> Thanks.
>>
>> [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
>> [3]
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users [4]
>
>
>
> Links:
> ------
> [1] http://vm.domain.com
> [2] http://host6.domain.com
> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
> [4] http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--------------040408000201060702050407
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi,<br>
<br>
If you have set up VM disks as Thin Provisioned, the VM has to pause
when the disk image needs to expand. You won't see this on VMs with
preallocated storage.<br>
<br>
It's not the SAN that's running out of space, it's the VM image
needing to be expanded incrementally each time.<br>
<br>
Cheers<br>
<br>
Alex<br>
<br>
<div class="moz-cite-prefix">On 13/04/16 12:04, <a
class="moz-txt-link-abbreviated"
href="mailto:nicolas@devels.es">nicolas@devels.es</a>
wrote:<br>
</div>
<blockquote cite="mid:c9c48e3eeb227b992b308c1241bb9847@devels.es"
type="cite">Hi Fred,
<br>
<br>
This is an iSCSI storage. I'm attaching the VDSM logs from the
host where this machine has been running. Should you need any
further info, don't hesitate to ask.
<br>
<br>
Thanks.
<br>
<br>
Regards.
<br>
<br>
El 2016-04-13 11:54, Fred Rolland escribió:
<br>
<blockquote type="cite">Hi,
<br>
<br>
What kind of storage do you have ? (ISCSI,FC,NFS...)
<br>
Can you provide the vdsm logs from the host where this VM runs ?
<br>
<br>
Thanks,
<br>
<br>
Freddy
<br>
<br>
On Wed, Apr 13, 2016 at 1:02 PM, <a class="moz-txt-link-rfc2396E"
href="mailto:nicolas@devels.es"><nicolas@devels.es></a>
wrote:
<br>
<br>
<blockquote type="cite">Hi,
<br>
<br>
We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
<br>
events like these:
<br>
<br>
2016-04-13 10:52:30,735 INFO
<br>
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
<br>
(DefaultQuartzScheduler_Worker-86) [60dea18f] VM
<br>
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1])
moved from
<br>
'Up' --> 'Paused'
<br>
2016-04-13 10:52:30,815 INFO
<br>
<br>
</blockquote>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
<br>
<blockquote type="cite">(DefaultQuartzScheduler_Worker-86)
[60dea18f] Correlation ID: null,
<br>
Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com [1]
<br>
has been paused.
<br>
2016-04-13 10:52:30,898 ERROR
<br>
<br>
</blockquote>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
<br>
<blockquote type="cite">(DefaultQuartzScheduler_Worker-86)
[60dea18f] Correlation ID: null,
<br>
Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com [1]
<br>
has been paused due to no Storage space error.
<br>
2016-04-13 10:52:52,320 WARN
<br>
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
<br>
(org.ovirt.thread.pool-8-thread-38) [] domain
<br>
'5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem.
vds:
<br>
'host6.domain.com [2]'
<br>
2016-04-13 10:52:55,183 INFO
<br>
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
<br>
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
<br>
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1])
moved from
<br>
'Paused' --> 'Up'
<br>
2016-04-13 10:52:55,318 INFO
<br>
<br>
</blockquote>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
<br>
<blockquote type="cite">(DefaultQuartzScheduler_Worker-70)
[3da0f3d4] Correlation ID: null,
<br>
Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com [1]
<br>
has recovered from paused back to up.
<br>
<br>
The storage domain is far from being full, though (400+ G
available
<br>
right now). Could this be related to this other issue [1]? If
not,
<br>
how could I debug what's going on?
<br>
<br>
Thanks.
<br>
<br>
[1]:
<a class="moz-txt-link-freetext"
href="https://www.mail-archive.com/users@ovirt.org/msg32079.html&quo...
<br>
[3]
<br>
_______________________________________________
<br>
Users mailing list
<br>
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<br>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
[4]
<br>
</blockquote>
<br>
<br>
<br>
Links:
<br>
------
<br>
[1] <a class="moz-txt-link-freetext"
href="http://vm.domain.com">http://vm.domain.com</a>
<br>
[2] <a class="moz-txt-link-freetext"
href="http://host6.domain.com">http://host6.domain.com</a>
<br>
[3] <a class="moz-txt-link-freetext"
href="https://www.mail-archive.com/users@ovirt.org/msg32079.html&quo...
<br>
[4] <a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</blockquote>
<br>
<br><pre>
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
</pre><br>
</body>
</html>
--------------040408000201060702050407--
3:06 p.m.
Ahh, we've seen this as well in RHEV and have wondered what was going on.
A better message would be good.
On Wed, Apr 13, 2016 at 7:43 PM, Alex Crow <acrow(a)integrafin.co.uk> wrote:
Hi,
If you have set up VM disks as Thin Provisioned, the VM has to pause when
the disk image needs to expand. You won't see this on VMs with preallocated
storage.
It's not the SAN that's running out of space, it's the VM image needing to
be expanded incrementally each time.
Cheers
Alex
On 13/04/16 12:04, nicolas(a)devels.es wrote:
Hi Fred,
This is an iSCSI storage. I'm attaching the VDSM logs from the host where
this machine has been running. Should you need any further info, don't
hesitate to ask.
Thanks.
Regards.
El 2016-04-13 11:54, Fred Rolland escribió:
Hi,
What kind of storage do you have ? (ISCSI,FC,NFS...)
Can you provide the vdsm logs from the host where this VM runs ?
Thanks,
Freddy
On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es> <nicolas(a)devels.es>
wrote:
Hi,
We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
events like these:
2016-04-13 10:52:30,735 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-86) [60dea18f] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
'Up' --> 'Paused'
2016-04-13 10:52:30,815 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has been paused.
2016-04-13 10:52:30,898 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has been paused due to no Storage space error.
2016-04-13 10:52:52,320 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-38) [] domain
'5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
'host6.domain.com [2]'
2016-04-13 10:52:55,183 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
'Paused' --> 'Up'
2016-04-13 10:52:55,318 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has recovered from paused back to up.
The storage domain is far from being full, though (400+ G available
right now). Could this be related to this other issue [1]? If not,
how could I debug what's going on?
Thanks.
[1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
[3]
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users [4]
Links:
------
[1] http://vm.domain.com
[2] http://host6.domain.com
[3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
[4] http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
3:09 p.m.
Hi,
Yes, just as Alex explained, if the disk has been created as thin
provisioning, the vdsm will extends once a watermark is reached.
Usually it should not get to the state the Vm is paused.
From the log, you can see that the request for extension has been sent
before the VM got to the No Space Error.
Later, we can see the VM resuming.
INFO::2016-04-13 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for volume
....
INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
virtio-disk0 error enospc
....
INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
Note that the extension is done on the SPM host, so it would be interesting
to see the vdsm log from the host that was in SPM role at this timeframe.
Regards,
Fred
On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow <acrow(a)integrafin.co.uk> wrote:
Hi,
If you have set up VM disks as Thin Provisioned, the VM has to pause when
the disk image needs to expand. You won't see this on VMs with preallocated
storage.
It's not the SAN that's running out of space, it's the VM image needing to
be expanded incrementally each time.
Cheers
Alex
On 13/04/16 12:04, nicolas(a)devels.es wrote:
Hi Fred,
This is an iSCSI storage. I'm attaching the VDSM logs from the host where
this machine has been running. Should you need any further info, don't
hesitate to ask.
Thanks.
Regards.
El 2016-04-13 11:54, Fred Rolland escribió:
Hi,
What kind of storage do you have ? (ISCSI,FC,NFS...)
Can you provide the vdsm logs from the host where this VM runs ?
Thanks,
Freddy
On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es> <nicolas(a)devels.es>
wrote:
Hi,
We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
events like these:
2016-04-13 10:52:30,735 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-86) [60dea18f] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
'Up' --> 'Paused'
2016-04-13 10:52:30,815 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has been paused.
2016-04-13 10:52:30,898 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has been paused due to no Storage space error.
2016-04-13 10:52:52,320 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-38) [] domain
'5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
'host6.domain.com [2]'
2016-04-13 10:52:55,183 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
'Paused' --> 'Up'
2016-04-13 10:52:55,318 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has recovered from paused back to up.
The storage domain is far from being full, though (400+ G available
right now). Could this be related to this other issue [1]? If not,
how could I debug what's going on?
Thanks.
[1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
[3]
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users [4]
Links:
------
[1] http://vm.domain.com
[2] http://host6.domain.com
[3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
[4] http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
10:57 a.m.
--=_446d2e46938523efd4a4a0e83ab083c6
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8;
format=flowed
Ok, that makes sense, thanks for the insight both Alex and Fred. I'm
attaching the VDSM log of the SPM node at the time of the pause. I
couldn't find anything that would clearly identify the problem, but
maybe you'll be able to.
Thanks.
Regards.
El 2016-04-13 13:09, Fred Rolland escribió:
Hi,
Yes, just as Alex explained, if the disk has been created as thin
provisioning, the vdsm will extends once a watermark is reached.
Usually it should not get to the state the Vm is paused.
From the log, you can see that the request for extension has been sent
before the VM got to the No Space Error.
Later, we can see the VM resuming.
INFO::2016-04-13
10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for
volume
....
INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
virtio-disk0 error enospc
....
INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
Note that the extension is done on the SPM host, so it would be
interesting to see the vdsm log from the host that was in SPM role at
this timeframe.
Regards,
Fred
On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow <acrow(a)integrafin.co.uk>
wrote:
> Hi,
>
> If you have set up VM disks as Thin Provisioned, the VM has to
> pause when the disk image needs to expand. You won't see this on VMs
> with preallocated storage.
>
> It's not the SAN that's running out of space, it's the VM image
> needing to be expanded incrementally each time.
>
> Cheers
>
> Alex
>
> On 13/04/16 12:04, nicolas(a)devels.es wrote:
> Hi Fred,
>
> This is an iSCSI storage. I'm attaching the VDSM logs from the host
> where this machine has been running. Should you need any further
> info, don't hesitate to ask.
>
> Thanks.
>
> Regards.
>
> El 2016-04-13 11:54, Fred Rolland escribió:
> Hi,
>
> What kind of storage do you have ? (ISCSI,FC,NFS...)
> Can you provide the vdsm logs from the host where this VM runs ?
>
> Thanks,
>
> Freddy
>
> On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es> wrote:
>
> Hi,
>
> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
> events like these:
>
> 2016-04-13 10:52:30,735 INFO
> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
> from
> 'Up' --> 'Paused'
> 2016-04-13 10:52:30,815 INFO
>
>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>
> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
>
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
> [1] [1]
> has been paused.
> 2016-04-13 10:52:30,898 ERROR
>
>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>
> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
>
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
> [1] [1]
> has been paused due to no Storage space error.
> 2016-04-13 10:52:52,320 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-38) [] domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
> 'host6.domain.com [2] [2]'
> 2016-04-13 10:52:55,183 INFO
> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
> from
> 'Paused' --> 'Up'
> 2016-04-13 10:52:55,318 INFO
>
>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>
> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
>
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
> [1] [1]
> has recovered from paused back to up.
>
> The storage domain is far from being full, though (400+ G available
>
> right now). Could this be related to this other issue [1]? If not,
> how could I debug what's going on?
>
> Thanks.
>
> [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
> [3]
> [3]
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users [4] [4]
>
> Links:
> ------
> [1] http://vm.domain.com [1]
> [2] http://host6.domain.com [2]
> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html [3]
> [4] http://lists.ovirt.org/mailman/listinfo/users [4]
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users [4]
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute
advice.
The information provided is correct to our knowledge & belief and must
not
be used as a substitute for obtaining tax, regulatory, investment,
legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020)
7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users [4]
Links:
------
[1] http://vm.domain.com
[2] http://host6.domain.com
[3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
[4] http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--=_446d2e46938523efd4a4a0e83ab083c6
Content-Transfer-Encoding: base64
Content-Type: application/x-gzip;
name=vdsm.log.gz
Content-Disposition: attachment;
filename=vdsm.log.gz;
size=28146
H4sICB9LD1cAA3Zkc20ubG9nAOw925IaR5bv+xX1hhRD0nm/sJYjbNme0M7Idkj27MN6o52VmSUx
pqGHAsnaCf37ngRaTbsLyKoCmrZKipCgrueW5+S58tPbWbAeUTUcfvPt1z//dTikmEiEOSIsI3go
yBDjPqFsOMzHU/fb62+GQ07pcPh6Pp3ZN2HwclS6QfjdPb/yw+GTN2H+Cp73TRjbD0+zi0U5u8hH
k4u5LX8rwzxDyF0v0HhUzjOMKLu9wPtsVDy78OHdRU5YboJ1iBQ0IG5VQLnzBnFTiEIL4ZjXF1dh
br2dW7hrbN8886NZcPNsun7EZDEeZ3n5jGMjMzddTObPSPbEvffZ99NJePofP6UhzXUTpF///Pz5
t69fD7Mvwmz2ZfYs65G/4Azgm858mY0mv0w2v08X818mSzjzD/NQZk/4gGS/ff0UwL4eBd/P8ABj
gg0mPCv7GRvo7OXXF+Uvk95/Zl/MXHw+/oQP2YmP5Hg4LEbjENEhih2Jh7O3wIHIG+TCZB5mF1eT
+cXI2nJQwusGQE87HgCDBqEcXk6v55dXozcz60bTyQUviBQmBCQFtcDwHCOjc4Y8z53DhTGE0Qs/
vbq88oeRAIZ3U0ySs6RY+P16OptfOBwKSxwB+giBeK40yr3FyBOlgiY50cQflFz7BEypBuS6v17w
X8id9bL5fbleGOc3yyV+/MNqwZxQLuJq0YzDWtqyWvbxXjfhfSNk8C0y+B4yVArMWcSGDFjF2r+y
o3E+/X3w+seXL6eTEcC4Ey2F+XBYrjC5XN87HComtuF36d4G99t309lLuDhBuLfJtBJ5oQuukVG4
QJzmDmlhJCq0zWUocppLdnFlS7i2/CS1owmAd0dk+wWI6lIpf5JTkFuCKQdabYpsXbow0oouDbT+
DdBr5pMBBvbe4T4hWHAZeW/oQO5Q/Hwnbjry/ETWW/jALWCFrHEccW2AzQE0OsFMasmV44QdSBnt
QZqYs7HejBETuciXLP4jE/9ZTiezazf49vfgFgDihdiBGO1jCboosmQ4FGZDPH+CQy/tBD7Olp+j
jC6ugczh9Rz+eZrFg89+lUY77gNDxjuGuGACGVZYRLgzQRsrrKe/DodX03ejyZusmE2vsjLeD7iP
QAS+XH+7noVrO4NLKsF/8f13P2yDHtg2nr75eT4al8AX2GL5UXlt57CcYIU+eT+z19dh9jR7tZhk
duLhRdM5SMcQaH49jZiUT8AEwm6hfLYSh1bvh4O139/PXoX5YjaBI+U1wBGG2b8XvZS9Sw+u7Lmp
D/AB97OedfOFHcOXn2aLAN/fge4DzOAAW5791wJWhr8976OMwtferZnrweExaM3nURvFU2TA47F3
djy6ufNjP1v0lMo91pEgWnnEpbfIUCqR89RSJ4KVNqTDhxPgY9Ioyu8ByAakEsAUzXFQAq7WZQUB
aSV8KX7JgeGLu/4K+EwlfCm7wcMyeLU1SZLAj/W1HL/RcoQYslPNrXRRXQ1XgEIr3wbfrd5u9Xar
92ir96R7lE+7ktuNys0yb4QCWPjpYubCGubh0PANR/DV3bODH95PlnuIWRgHW4avxuAXLI8Nbo/A
puFfi1DOy+zfH7Obx8cvB4JP4/3wOTtxYbwB3qcDm9C14LkxbDfPfXCzUNRkN9yR4czm09k8cvg7
Oy5DJZD//dWr7198vxVMFSXn6sV30bIAnC9/eLnyAqIILsqnGRzIJtN5Zt+BZ2fzcRi0e4uSGy/5
W3m13ERWvKaf/e31y6XUltn7UfSBQnY1KktAdlDPnaV9FWN9nZt/ny5KnKObL/gqvKv1QLWN8fC+
0u2Q/HMyn/c1bbcojsR8qskyvmdwZXC/FpISkDQd86voEsMm58d8bAhdB/haR3d1XyvaMb+CLgaf
p9qHi5Zq31QGBm8CnXQ3buo8k1SjclpepPith01R7SGWNifK6jD2KasDH+9ldTTDiqyyOnQH63dk
qEnMUDeKcT/itHxEOuq48wjsU4wNXaXlebO0PGmaaPxs0/Ik5jDPk2LnmJaP5IrFO2eQlqdCcSW2
Z7ITmU/leeTlNdea7dfgO7KUgI2mjZTZI07NRqSZOhcNTgSmaqXBq/LrdbaZgJghneNdSRfWzic9
zvYbKy6X6siISn10L+i3S6pJH8eg5+VlTJNfXg6HAsPXm0f8F/z/6tq9DrN3ywjsZRk/vVoFW59m
z+14HEOavTU1fpxOx4MYkby+WgUle4Bkls9G/k3I3o/mb2PKak3Pa7j2xTe9YUwSJUhBryKum4BX
81g+9qBoMBhC5XSOeDAWgd40yImCca+4pSS0qzfgW/P9S+ib1hts0v9Jef3zzy++eZZI4362p0Rh
J8gEy4YlCpsgV1Up9Mrrq8ty3lt/fBETPKqfxc9rORuCDP74src69ncQ1njF36sSQLuFZolDs/Rt
osBspG8fBK2TroVaea0EFB40r9UIvmPmtZJ5XjOvlcjutLxWApAbxoelG5+1mqhhe+otsvuo7Nrg
k76IecTD2tEXk2J6XBO6DyXRSm2Aw5sbjxH2CoAyVCNtrUUAq+POOqIVb2dC8S57tIS+sQm95UTk
wqEN6T66V+kSYsx2XbKWpciJtQjGFVJ+b6/Cs19vbkqB/Vd4EiD6K/EFsM0GFIILCNhlkPGSIJ5r
JaguRNAGmLd+b/beltmV9ZF3WW/pD5RvwThevPPl1cVaXi/elleD6w+9DIQ9ZD0qOe1ldp717tO7
dxiSCS62kmxpGN6Mom9xcwYEd/YhyuJ8mt2c+2QUPq3OJDL2smI6y5YuyfzDNdy8JIc/EF4yVkbX
wetVQyRGJazLEAbgP71fYhOpA6y+QQc8lMy6+ehdyBag0w4k6ZQlSfqbmZ3MTyPmf42vCv7GHjdC
c6VIdQwT71CkN8T4al0GVVOZXn7aw9w8YJjVoUv2ZMXZBpxUTbcbaZjF7QbZu93YbRGUaeGh3NVQ
lX7KKKqu+GECAhk3Ft9Ps29C6WajpTmIe4xROQWDVox+X9rtY6anRlcAbnlB1n9QxT83fyJgcTdx
WX7aErnpZAKowxqPlXY3u6JYjTe9sqNJuQQ/JcI+/GqpH/opMN9cmxLyu7k2JcFzc21KiHt9bUR7
FXm6XCyWJYVptaJ36hl7bL2tvBzd7DujNYhnXrx+/vrFxktWFKYfVxS+EaQaFb6fWLfo3WIA4v1b
VODxjVIIrZjGdFUeuymIvXjAjsNsHh/wP/+7vnM+nS/rNHtwGzexq4fquwiyelW+e2HkShiqpOJa
n/Vi2UErQRXBWkrD+V1a4XoFx/tpxZQRlHMJf+sylBKjJcUSU3hrFUuT6473gglIUQqQYuBsXTCJ
BEEAomjNNa4CM7n8eD+YhoHDgLFpACX4yoIbzDDHf2R5VaxmjyGNRqphCCrNkm6GoDqL1VmszmJ1
FquzWJ3FamyxYpD7AcKEtbILCSicLruwqBeGGWZffIrgvAI3uN7d/WxU/iO2RQHjY1tUL5vm/1xa
+knofdmM26dNdDQBSJLtMaRbrlVH/panmgb+DgQ+07XAf7U64BuHK5/gzUBeWT/+U40G3xOovIdG
qzhlPyoAH/k4XcyzUZHZyQeQ8nj6vR0t40YxLDuaV7RGNUJP0Frowea6ApyGLOtnz+HxS/23LiFp
gBU5ftBuf44wAciD5gir82qdL9T5Qp0v1PlCnS/U+UK3vlCtklHaN7HMoCulvU8XfZZtjJxIte5k
a9vATHh/uYfqmP8Huog+jr0P58d80DKrOmrSvohe9THrmF9JF3mORfREqFUDs9YDkVJEv6M/E5Ck
slWBbR5czgrY38ggNeIKM2QLnyPNmLWF4VgR1a5Sjm6vi1hDf9zhdnver/4sw+1WzWL3phPJgTiT
+VirDt57AOoBO5P5WMvWpgoCVg8YO/18rGXz7D341Jb5Yg8wH2vZ4JckgRWh1n1qTjXNTSequAee
btct3275fhbL96S7lFqJugQUHrQNqBF8x8yOJfO8Zog/kd1pIX66a/DcEkxzgPF2dd5ykPF2CdNb
iO4bQ89y8MP5jbohpo9Jk5/7OPyoG8Zgf7EcdQOrt+LnGGq5wYCW6gKDVXRZdgCeX3gAXGK1Hm5n
ms05onHkj/jM5hwB0pQ0+tWGo8w5MutRJ2znqJMdY2ho08Etn+2co0gx3uT3dD7POUeRXPJM5hwp
zsyOX+xJZL48kzlHhGm8ntrU8CdoABvNyWc25ygiLeTZaHBsVkysnlRXZ69BSZ90c46q6RIV9vnt
wQiXdJ2fq9qD3fP+du1LaJ9q9UjnHO3DS7cJ6gjmpNI5PFYG2DzYuHkA1qACq+AIwMdcyzlHamvq
5wb6s5tztBNkFsc4nv2co91CwzBtGshPFJgjzTlKRuuka6FWgDMBhQcNcDaC75gBzmSe1wxwJrI7
LcCZAORjmXO0y10BVPQjnHOUgFJztaEot5bBY3NmJeIKfMk8WI5cDs4QJZ7wArczoWSnPdJtRgXe
nWpxaEO6j+7iwecccWeMDSSgHC5B3GO4yxKPgjDeM5sbZ9hJ5xw1IdljmHPUBK9HNOeoCXonnHOU
KOZ75xwloNluzlGiMj3inKNkFGtuNxIxS5tztNsiGN3NOeo6pbpOqa5TquuU6jqlHmpqxB5DanTT
EFSiJe3mHHUWq7NYncXqLFZnsQ5ssU4aJqyVXUhAoZtz1IJUx0x07APInPeco0bgn9+co0ZoPJ45
R43QO/s5RwlYHTtotz9HmABkN+eo84U6X6jzhTpfqPOFznbOEWV9Itq17fw5S2mBLrHV7xxLaRXb
UUpbC0nRp7ECsmP+fbqws6yjBjWq1syv6uuohaTqU96t/Eq66LNc+UyIdR/jrha4HS25NLbkNulQ
+hz7lyOx2KkaoHb3L3NCDdbLEVdYtuxfBrSo6XR+FV3Yec62Y4KrhHW/o2qZxVbez61/OSLN8dl0
v3G8bl+ustvXYTaa+pG7WLaZ78CIk1blzaBNjQ7gs3BeUFCwKke5JHEBUm6ctsFrWqu8+S7gW8rY
lnC3+AHXf0zHi6vwevR/4Unp1zXNSe5iP6tbAz26erO+gRPr8uVjKWVALyWQZZigIIGIxAVw9OIN
76bj9Q1gnYOGx6FCchN/atkh6wqMFLWOem19rvN7VdYN6VevDPCWfpUVgPPZIpRwcuWUGiq0Ukou
XdDrWKoxmVec/VhHZkXT6pBEed2sDjkhNiddgbsyh/WAP2VHUlvIDpiia8LhmhmGROZWZhg2wDN7
wGs1nxQT57gtAnK0IIg7gZE2yiJRUA86tVDe2KYmwOxWYU0nk24xAYnR8eYmwFgB5oQ7xDiodU41
QSbHAWEgU858nudG3DEBITivCo6Rp9EESCBwDlKArPQhp1bnLKhdJqAO/Y5mAhiYN0phz0mrlObG
2Y+NZLb2DyGmyes2E3BkbE66AhNNQALwD2QCGkF2HBOQzOG6P6aTxtx9JkDuAa/V8EetrDZSgv8m
XI44MRZpKgqEifBGE14AeZqaALlbhalHZgII6G1mWY5CjgW8wHJkRG4QyR2jnOjAi7smwFHrOfYc
uSCiX+IIslQYBDdJIbCULhe7TEAd+h3NBHBquJGKmsp988bZj41ktq4JSJTXbSbgyNicdAUmmoAE
4B/IBDSC7DgmYB8ouqEJSGTuPhOwz0lpVcCqrQnSeYFszhzotahClS6QYiTXFktLFTtSIKjFkJgH
MQFwiLnCK7DoSiFugJPgFyikvLRB8uCpD3dMAC9s4ZzyiOVgOLhgGulCOlQUhcMCe8XM/Xb7hvQ7
mgkg2lAujZFcV2nNzdO1XOvGjUKJErvNCBwdn5OuwgMFg6rqQ88jGFQJ2cMEg5qWmyYyt20wyLTq
Y1DeKyUViiVPiCteIBMBZdRSHgBAbuqNO0kPZphHZgacEtZT4pFSDG4IHF7FDUdCOWwKqUEVqDtm
wIucazAcYAFyuIFQj0zwsWaM2ZwUysBrWgSDzGk8AdgeEy3wqoLz3t759mwd55rixt2iafK61RM4
LjYnXYGHCQbRk/5Wel2ynkkw6JbDdTsO0pjbMhjUUgAZ89460GYhB+XkwGXJOQlIWulcQTkFLXec
YFCE+1GZAJ1TragHY65cHDLHGMpdIZCxRcAyd9oxfscE0MCLEJRETgQfO040ygOQ2nErQ9DgDiw1
UcNg0F36Hc0EyFhdDvTDuEppbpyt41xT0tgEpMnrNhNwZGxOugIPEwyKwJ9nMKgasgcJBt1yuK4J
SGNuy2BQOwE0FjSmygNSVDDETTAoL0RALAirmJXU4Xo/WZkczKDkkZkAgo0pmDNIiWgCZIBbqRBI
Ow22AefW6+KOCbCeSVVwDhBRhXjBgP/ea/AdOLMuSC+WmcmGwaBY0XYCEyAMFgI2Lqwyibpxto5j
TRuPLU6U120m4MjYnHQFHiYQRE88p7gtZA8SCLrlcN2qoDTmtgwEtRNA51lBhWTIGAJqCnNQUzpQ
pKUARUiYx+FIgaC7KuwRmIBAZSGsMKiQGlyBHG6w2IFnZyUuFM+ZwXcDQbnQ3BIfUGDEIM5hH5B7
yxETPPcMG4/1zpTwHvqx05gAgamhUleHzzfO1nKsWVMTkCivW03AcbE56Qo8UCCInW0gqBKyhwkE
sYYmIJG5bQNBrQTQCCxpQR0SeQAz5XIK0NocGcK0Cjl3XjcuDN0TyGCPzASAU5fjJSe1BVJZL8EB
MBjJoHISpPEh3O0NCJrlQYC5IDYHtwG8CGS04UiqwHisQg/KtggE8TPOBeyTWd7YC0iT18PmApKx
OekKPFAgiJ9tIKgSsocJBPGmXkAac9sGgloJoA0G51oR0Eu5jC1MBcqlx0gJFkC1hVD31y/SAxkH
bg87vhfgYIuoVYGCVeAFwOOQkQQ0OhYGrrbe67smwOYFGDpKEZEKgwnQEuU2TtkxDl5hCu4VaREI
Ok17mADqKbB8tDJ6vnG2lmPduD0sUV63egHHxeakK/BAgaCzbQ+rhuxhAkFN28MSmds2ENSqOUUJ
E+dhCdBQPlatKI+MYwpREoIz3BtnGucC9gQyHlt7GJc5tiwQeKIAdjoL7Iwdwtz7nBMjnQh3C0OF
CEwoMPyU2lgRVliktSdIeMKsz2WupG8RCDpNexglXHHNttRRbpyt5Vg3bg9LlNdtJuDI2Jx0BR4o
EHS27WHVkD1MIKhpe1gic9sGglo1p6jcOMVJjkiOo5rSRUxZ/z9719bctpGl3+dXYOaF9hQg9x0N
1SpVjq0kKlu2V3KczcZZBVeZE5FURNKRpzL/fc9pkBRlQkQDICnIhqdmhgJA4ty6z6W7v+N7SZTq
JIJfzHRtF1BSyHhox8MCHtMk8iHsz3BbkJbCC5UQXoKr+74Ioyi5nQUENA2pJKknM5XhYYLYg6hA
enEWJllmSkdxg0LQbo6H1Zs0rW22sguws9fNuoB2jsANFYJaezysmLL7KQT5dV2AnXLLXIAuIa/R
wRTFk5BSwrxYAaEiJsrTPpdeppiMwftlcf3jYXr9FPbQjocFPssIyQIvkegCOAm9SNLEiyJBOOeZ
Sujt5WBGqWJRJj2fpwhDzrkXRoyCzwDfr4iImVwLElFFfi1zAdY2W9UFWNrrZl1AO0egpQuwIP6e
XEAtyrbjAqw1XNEFWCq30AXMke/WNCAA2nzltxLmc9A/vwpjmNie2MDMbxTrk5P1EtNBKyWWXl+C
/p/Y4I5vFhp1vYFpIncEjSrEAhoVPn4OjcqEINpAo9I9cTc6ZonyNdG74obccENWuQnYDOSXrsX6
FOu5UbVgL2thfdrEVxsyyBKmdWuwPimTkuZgn6ohSDenLkf82Q6td1UuKmgjWi/TLJih9eoC5a80
ElqDQM2ZKzGwWGokROwbCT0LLy4wbPi8k9Dp5eA0702x0k5o2pvJE1vzHD03zSusEqKCBmhlfDXa
qJBG0o9IIj3OaYinKgIvUBBBSCEywpnQoiKCVRH5d2VZM+rrZqkL+T+qmnTeTgmrkaxIg9rgguTC
vBC7DY0nvdnHo3nfofHCziCHgjE9a0z0ct7Zqahp43qjMTzUSxAtDWY5QbwPtnY6Fir1orRgYZf5
4mbo22YDyDKC6uaOluq262JnQeRGu9it8T3VBtkqK+s2bQAruhFOCsS4fhZlvidCrSDKTQMP5uvE
gzQ7lgGNI6WTZv5mzfauGfW1/A28cIScjB+VOI8q76/gPBbvL/IcFXrNxaMEa3EEy3PxZGraYGFj
2NsduszdP6YQdiY39xMM9NGOlvJF02IQItdnGBEa/Ke9wDTSwp6zs69WazRnRSCxIDDv9bFCIN/z
Cwm07u62KQnm2U2BAHUhfdZt3TZGH3ZaWCGP7YlC8qzbuW1Mv3mKb2WATWa5ygGS3Qy3FCB1o7cb
vd3o3c7o3WmMUikPsGDhXvOAWvRtMw8oISioiy1lqW67PACI/OnpyaujV3eSSdFyBkffoWcBOo9f
H+el1DxMf+zABWc4mjjhx7B/EUYXaUFj7/K3sMVbcPvF4iUvxiYdKHqN67w4PTZWO4ZEAgvJqTPo
j8fAbAEFfK0yfCU3XWwrbtu9yTqbBUv155QopbEfyMRLEgIONMWDjkkaekmQsiAIOQtS2izv4evy
DkN97TrbjSZQC5uutpXJvWizKsW28HdNNDNbQk3MTBDT6PGrcJAe/Db/kg3tv8EvAaO/YQRBRUg9
pSWEZ5JyL0yF9LjWGREBfPRxhpi91/kzHDuDMEHdOT2zaDD+AAHik4/JePBkZq9PPowHe5efeg4Y
ewp+WQnWc8KJ01uVd28zIpNC3iky4zXO+7gAMb8Dhnv1CW1xMnLm9xYeYzE6rcTYc7LRlWPWLbCF
utMz4kg2xJciohpfJzWZ6I8d7Di957wa/Wm4QemAqufsOI+oE5oG1c4U5rQNWTrjVpZ+fhUOJ7sx
8+/xVWkyd9a12MwnUo1b/9ZMpHNhPJ3FkhUn07NFgDP/gX2nilycR7lma2iy7pZGS84wFqGlsch6
j+A3wL39bIYqXMzo49SFH4ZgkBjavxo5z9NxfNU37uDzTu3T3jZb3fYHQO74CZ398wr+Z/4PCcNo
4mzRdx6SnuEQWE8Tk67MS6eY0owGYX+Y96a3yeH38wb2rg3N82dtMtv5szZZ5vxZm5Rv9iyynS9P
n02nJi+z3Q26lBT2+Kz2fNafF6fRG+Cdo9Nnp0dLL8klzP6TS3huSBXKJAvVTXs3HIB5/44TOL5R
Sal9rgnLQQuWDbFndi5epFcT/IFffp19czKamGS3B18TAaVEMn2bQV6tUlJKo/AlnnDxRV6nae1g
WSMryRAGVKlAiNuyItWKNuWy4jBrMiEU/KeqQhkNtGJEEQZvLVKpde2mlExgijGglIBmq5JJlUBA
VV9roUkRmdY1nHIyAy6VJCSoQSWRUoqAcCLI5yovKuSUOFJdG+Dc0pMur1N3HqvzWJ3H6jxW57E6
j7UBj7XTMmGlpQcLFna39DCtVobZd/5rUcE5gTS42rddpz9+h2tLoHhcW+o5o+hfxtMP09439bS9
21WQOgQpencN6UZrxZU/c6tu4W9D5HNdifyT/EJSu1z5iCwX8sbV6z/FbIiSQuUKG43qlC5OAAnq
cTSdOP3MCYefwMrx9p9h39SNsCzbn1Rf3SlmT7JK7EFwXUBOTZW5zjP4eTP/zfaZ1+CqbmeCCkW7
8gVECyI3upGweF2ty4W6XKjLhbpcqMuFulzoJheqdK6Mu9z3u/N2BXIR8KmF5+1owM1Zy4DuyaaH
LWXeV7NT/qpcpGij8plWenbY0m+qfN+FebxTfoFcJG12AnkrymdKUWGOynNdOPLnx8bXnXEKXMLb
CftgHzltFPShRFiK7ggmgfMFTAJ8/BwmgTMM8nKYBNJ03AeuZN24L5QL7nlq27inWnAqUPdSFsIr
zE3Zv5s3QVwmzI7mmRDgF5d4ml99dAYR6FU6/vDcpNxYjDB/Yh0kz8IdmzDdmiIYfJcYuoK+h5Pn
/XiCO0aXylNvPru5tFH0GbwBd7eFzs0vOAk8lVdGXn48nk0PJz85URj/ng6TKmQtcDU49+9SPv4Y
bgsdV54el+ar8e/9ywOC5suI0AuUERsx38yAa+a3El7R4CvyWsPIDWszC2dg4asAIlyyfHYrgpCw
5QXuD4BquK3obUaAA3NUBUzm+fNvkZDHDlAPudVnxGM29Rn5eKkyAz3I2RBObd/Br1bh4PMBwbhY
MyBm4/Wxg79vNikDhcvG//jgl18bvV6uG4+L1x8OLiefnLlBNnnhhiaAd9+/Dc/rzwFI2cXHwexM
xJyc1/CqEOuqx9NJeo3k9Idzu/p4Dja1eAArmAPn5q4zmt/pOeejiTP5sHTJGeDP1SBNbpq0q/mS
TBP69Jw+aim6i4+7Et2CNGYpOnvSNio67pN19MG7RmFSpNb8zjZExgJyl3OIB0kVFzieJiPHG+YX
xngFKQdu8CujYdY/d3oOOMG+WXfGJfMM5uo0OcM1jjF4hF/+8X/GSQ7M1ucn//jV6Z8PR1fp2Xg6
voRBDo/Ovg4+8c+r/iQ9i0OI1MxiQwreNumP8SjVWZhhIRx+fHR1lvtQ7kC0j0co4C1O732vF/61
/CquIFZLIx6KjIZxFDNy65/gVZ6Ogr/wDa55z9Vfe/80fzq/Ov9xnPOLURReAPfzEwxnWMYHdkAc
/RGw1B+nZzlr+AAyatbnzjJgZXFlCg+BbHEWBK7hZ3EWnF7iz17Bxf7wbAAh3YEjyeJCEn5CEePT
PdDHcPQBrATej8qZwtQ7diJzeTzNsv41fBzjhoYQ7MLp/YXfyFWBYc1lmsQXU3MgxRs5uKbhogbd
cDK5ctE5umb9M73GqfvMXJh9Nrowd2cfJ+H52P14DtlAmD84+2x+4OLj7KnLpQ/4JqtgtSISoKiL
nffVQk2ixFg7JdZGqEkUF+5AaAHUJGdyVnOg9YAmDS91cFm3ADSpiWqGM4nMaPGV4UwC0wFGwO3A
mSSYaaESRaFFVqn2COpKv8OZLJKLaue6F/FzmElWCHu7skFn3bTEXUIeKsxkGV+NYCY5SwLtk9AL
o9j3RBJzLyBCeTzNMp3EIvNFQ9gvcudhR0N9k+Pv24KZXE8ybdB/YGcwkyVGQ2v3IbA0mC3BTFqz
tdOxUGmPtwUL9wovU4u+bW6sttZ5xd2hluq22x1qQWRLYSYrxQrcVaxZrPBlxlAgl4Z7qrYSQ3EO
DlaZHQTFy8hV4FKByQAByR8afJAFS/V9BsV5Ioqlx4gIPSES4kVxKj0aKpJQmsIUE28LpntOfUvh
g0rk7hedVdgtfBALufBDUFnGE1BepIUXJcL3MvALoWScK9OqcnfwQXVE9hDgg+rw9YDgg+qwt0P4
IEszL4UPsmCzGXyQ5WS6RfggaxYrxpqWnNnBB633CJp38EHdAaTuAFJ3AKk7gNQdQLovMIYSR4pO
ql790dKTdvBBncfqPFbnsTqP1XmsDXusnZYJN9XBbM5CBx9kJyrRomZqdxLUIvigWuS3Dz6oFhsP
Bz6oFnuthw+y4GrbRbtGfQgNkbKDD+pyoS4X6nKhLhfqcqEWwwcJ6SrdwQcVycWXLdwCJCSFNAG3
ADHWGEFG+HkLlU75K3Ixa95tUz5XVHFulK/3uM3+rzUol8ikaLRZKgzTQAWxmk3ZhDIvYvA/hFEJ
06MQLM221WttTv12e0yvf78kX0qPaTwoZsLu5Rah/l7uXO6/R22OnrNCoL6jB+zue9Sac00r9Kk7
mvy2pEct9fdUIX330KQWD/dZGWA1pO75KK23Nmk5v91zh+lu7HZj98sfuzuNTzbV5GHOwr0eAKpF
3z11Vrit84r1XUt1W8PD39352ZDJNtBfuvwtfLP9pS1QEwWiJtbBL/gaISZRWFjrbwPEJOFSqLvh
HiolwIGreVcYKJJLQFpYGBBSc6ZmVaGiwoAFbomsi/Xx1QLjoMQUb6XE2giMg+JCJ9YCYBwmhPZn
aDJFYBSWyvfrIDFvARrHV0Q3gsaRJJ/XvipoHGSatQUahwRSBzNonKIJvIqXktQNaDMg5C/Se6Nc
ghZC48AA0HlVn+wF9eBzJXNh5FgiPa4y4TjzNMSZwQ3ihp+r6XCIKVI0ncwRF5M93O80Ca8miwej
NBtdpU46hPv4dH/y93rkz2FRBV2L7VkFO7Mppuecrg1g+n6GZ/v44Jfes5dPT08PnsPfWCp5fnj6
7OTozduj168OQGfjPjHrwkevX8O148PXP749PXwGQwAvvjx8enp4cvj25Ojw9IAvruBz+JDKH3r9
7MWb1y+Pnv18MP/z5PDV4U9PXx69ent48u7pS3xW5ve+//bli9Oj/z08kNS89s0PP39+5fXrl2fm
RLnlVq9p7807cnD5cb8URtPFDS/72Sfm8//2/h19xyGtY1Mv4pfEe+6fn3t/HvePvezp8ejbU/cy
t799Ap/M3LBPGdPSHYSXoyyDuWrfcH/y+uXhwUl6Pr0Ir/Dv0+eGdqv6HTz+9uc3hweLLTPvDk9O
US9G1O++N7/0U/9p/91b7yh5c+4l/e8G3otXf7z1/sf/cOT9OPpReH9QFh8agI6z0x+enj17cfrj
8QFTKiE+zSIFTo9zFYIcuS8IEKMkj8KYJmkSM793C/P5jjUoY50a5jMY2UIs+a7Z/+cg8PmYuX26
Gg8fjy/DGGG0F/sXh/OLVrCbZ2YDy6tTJ7xAOj8tTp+nSVXSJdkx6R9HF9NBY9pv4obA3zEDMNuZ
TSVmOivlY+3sFuwMHhh8+UPHB24/4m9VxNxK8ROH+KnrNrUqF+Ey3r64khEN//KUoiisXKlGr8uV
hCuDjYMF7QhxcT1fMOKbLDJJHG8x0WCoUQRESeXpSIYeTVSQciL8MGXNNsGIuz3RjPrWIS6uJ5my
B4C4WGI0yEO9XQWWBrMlxMUythodi6s9FiotuFqwcK8LrrXo2+aCq7XOKy64WqrbbsHVgsiWIi5W
AY+U0qWbRy7ePuheGUuNQIszQRM/iSOP6ARi1izSXkhj7kV+5kufUpnE6bZAi+fUtxR0z0Lu9wy6
l9KUBBkNPJ8J5Qkdx17IUh+P1sUp9TUF/7hT0L06InsIoHtlfMmHDbpXh70dgu5Zmnkp6J4Fm81A
9ywn0y2C7lmzWDHcsOTMDnRvvUdohAl/e4bqQPe6Y7vdsd3u2G53bLc7tlsRwqjEkdZv+mHpSTvQ
vc5jdR6r81idx+o81oY91k7LhJvq5zRnoQPdayCqe2otZQgK2g26V4v89oHu1WLj4YDu1WKv9aB7
Flxtu2jXqCvbnMgOdK/LhbpcqMuFulyoy4VaC7onlcswbus20q7IhTfEo9vGRlpKBFi16V2+p5ue
zdMub9ha9MtUvXYFazYktnI2jynKA9+0XA3Wnq5dAxIhESRCt/KcePsQNUBYEIK0A1HDl4qaca+1
7/zeYNyzfUJc0SFqFMpF4t7Oto17SSXLxz0N1h7KvRPwwPCmdR1T/koRNYzEAkJbKbH2IWrk4qJ1
IFs2j6jBmeaa5SAURZ3JLZXP6ih/C4gaAfCjGkBqzLhRXxWkRs40VnJbAalBqRbKgLzwPdkobAfG
mCuRlc59r8pFNTsSup2wXVMujOpVIcDPSmlZruGRuxRxi+qvXDIZ0FCr1JPKx83D4Dl0BAF2SJT2
hQr9VCbNDjjIO7az3lC/XaDskvf7Xw5QNrq5FbRTtsfbArdrcpYVAmF+bAncrpmUCwTI2gy3y9kd
cMD3ALdrQhMrCyxYIS+b5vy6Wwotp7h7x8ruhm83fL+C4bvTKKXS/ioLFu719HYt+ra5qcla5xV3
Zliq225nhrwbyDonEyXUGC67/C10s3DZlVIh4Uq/Q8cpkotSzdAot5IiqoDLvLpPhfWy3uHJyeuT
KkwqthRGITff4i9eDvDj4lcfnZmJHybO4/x7wPKb47PlZ5zZLzrKMbIZTwdOBpeQGbTneL7Fa/ac
e3NlmP6Jv1Qwnu7EBgOGpMvxBOqDBPwp46vRNuQwpSJkEA/qJIKYIQwCiGk4RK2c8zTwExJq2iyZ
vwt67ob61gH+rCdZ0AcA+FNiNKI24I+lwWwJ8MearZ2OhUohowUL9xoy1qJvmyGjtc6rdlixU7dd
yGhB5EMB/Fm3RCfdIJAPD/CnjKVG9fA0iWWSwc/C6xNPSEo8rYPMYyJLhVKJZLQh4A9d548M9S0F
/LGQ+z0D/kRRGNEo0jD+I8gbOcu8gKbCCzKRqCBmjPhkp4A/ZSIrytgfAuBPHb4eEOBPHfZ2CPhj
aealgD8WbDYD/LGcTLcI+GPNYsVww5IzO8CfNR5BuYSKDvCnOzLUHRnqjgx1R4a6I0P3BZ+wzpHO
nFS9EpSlJ+0AfzqP1XmszmN1HqvzWBv2WDstE1ZaXbBgoQP8aSCqbS501CGoRYA/tchvH+BPLTYe
DuBPGXuFwOutB/yx4GrbRbvyNUILIjvAny4X6nKhLhfqcqEuF2op4I9xVqpDfSmUiy9aeHxUaKYD
c3SYFR8Ar8qkLGoPqhSz2Rv8IRwmF/PQZVywNfh4sTV4vo0YuDBSmm8I3gy9PtkMvZdXI0h2x3l8
mq/Zg9auJ8Pk/TXl769j8f37a5hq4S/y/jpN4Q/4bxD/gLfeX4fB+2sCN0K8KX59f53AB8Jf/NM7
gYsMnozwm8/grv/b++sofj+Bv5Or5Ua+1NfLf/b+loVaR1HAYLKJwB/HYQrzt048mMP9MMkCX/O4
RGS4zG2OyGM0mS8iL8LmpeuPYPCeTIdDYB8jZoj9p4MovXJGmXOVX3bMEN936N8UgeA6I7GnkgCp
4pEXpanykjRIWJYRmep1x0OQKn2bKhqIIrL2fhpd/Z5e5RdQiTMdvUqvJzmVJtp3bAiaswFDMpsO
Y4yQ8RL+AG7qIdc0DQUV5BsTXu87j/LHgIrx3mUI+UB4gYUIhCYwjyNSKGgr8INAf+M6XEv3vs3l
8aY0Ux8MG+YoFUYBzPKxAkfOIg+cuYAgMoiA2pj6gaq0hW4zDK3MGVSopd7ob473jsGqMCaDJG+S
DhOzVmVMbZGb3f7bdS7DTxcjsMr7niLsJHT3JpM7BKRoMwHtO6l5EPWcN7Z3fEYTEoCjVyT1PcEh
Bw+ikHmSMQiORUR4QjDLzXM+xyZ2dR5hIunYBBOPsVY27v87hdmCqk0Y1q561EM+bET54PvUC972
PvWeF04no9mXhvDnwmAGVvb4xMrGC9Br1m07164gytLWVuNOx5kfTnRmksF63tynR9PJXN3JHpYz
J+DqFg9GKUg3ddIh3Men+5O/FyMOWZC/I8Qhm1RyQ4hDJUxz2h7EIaI3hTikXV8VRuFfecqoXbNT
tnUpI8dyiEkZ+Z7fXPnFTO4wZSzF6gQyEatTtBKur23Aprmw+K7g7UqATSnnOke3ow0tlRJXd7im
hXIJKG3fNCUJFzKvbJE93lz5AS1Cxdv9NLUOWJLOoBhbOE21EVUUxYXHo1oornuErV0He4kSU7WC
0JZjfZaNKn9n4LNr4VqBbRKY2CtgogCo29Lo/ToY7duA0vXpDHq60D1bGqRuS1ZEAgJWyZEh0Tgt
oiyfy7t4Y1UuWNpvW7whuPC1P0uL1AaUX8hkS1fSqHQJaQZu/2Uaq3Rpw9rGVvq8cBqYtMhv2OZl
DYc7tNSV3XRrklOqXMGDBwr39P/sXW1vI8eR/p5fMZcvsgO23O8vC9hA4CC5hbP2wTaSD1mfrt9m
zXhFLkRqLd95//tVz5ASZQ3JnheSIy83iCHODDlV3dX9dFVXP7VHL457bbQ5SySIgbCSGHHsLbIc
O8SIkkJILmToyd1Mt24araUfHd3TbpGFfgZ0T3uM5mF3tu1Zu0yDORDdU7ZaRx0LrQ5kZKhwUrqn
TvId8hTEPoE6127N6+68VO4MIZ8L3dOO7WpQRWL9/Oie9qlk+kwbhAsVfVBIlsQgHlmJNPiESEeh
lOOeUo8PVf5gLf1I6Z4y2v3EdE+81JgrZ5Dy2CPOQ0SWSo8cl5zzMmjm3VHpnro02XOge+qi1zOi
e9qjXir5d0q6p0wz30v3lKFmP7qnzMn0gHRP2Sq2XG5kapZH97QbEag40z2dD4ydD4ydD4ydD4yd
D4ydijxjD5BS0TUElYmkZ7qnM2KdEeuMWGfEOiPWwIh11DDhUPXH1iqc6Z56NNWJSqFtFWhEdE+d
xB8f3dM+NZpKFDwjuqdO6o2e7ilDq0MH7bKqCO4T8kz3dPaFzr7Q2Rc6+0JnX2i0dE9ETYg6n91t
aheKR0j3RAQYZnVAQcr+JyJBSTK21N8dJ+iJnvB+PM1MhTIqmFaMKj1MhsGlItIO0eBtdEFZLfyh
KqKupe+U3wMvnFd1oT/pXN706ftb7Cbfv79pE7nFoiGrOj2b7K9OXx89qtaKm9Xp1WUNgpvV6Vut
GLIExDkCppPLT+Qzl6xRvmyUHqwB66NO2Q2Yjc9DCVgzVDyRj8C9RgGzkXm4Hq7O8mW1YLvKp9VI
7bylljnHbWypncfvefyex++hxu9R1ylDVateqzDWatXb5TtRterHfd4yNJnZ3dnVqtfsXlvEVMly
rl/+NWELyPnqm1f14er6dMGnBVwoZvNlYd/DCj3RgDVEadu8RcmNl3y1qE4xNL1mUnz13avKahfF
z9N07D8W19OKfvUyh2ErvTaFWj8qWrGkdCJ0HcUBeoKlIbqmFWs6mJpBPkQS+RAbJafH+JiaUmMl
VuoRMDVRRTWtqSD60lCDVpT0C778PoNSZsLl+JgTKNVc6IFIM7apeAI6uR0cLHTFWjLCSWqkxEOp
xdJC+KMiHqJkgnEXqp6Pk84rNRc9Fu/gHmYjQkUdZ+/MbFRpQ0eyMIPbjAlZT9JNZHqZ1kzHQaNF
CfgXsu6evlQ9oBZXZ+qbpnYRZoS8kIZJsOWq7/Gl6d/5zUoecb3Rj3aeUlgsq/FQgW/84Pc/wk/d
o0Yd5kq//vU331ck7DHAxcd9N2BbVHQG67b45l28sWnj7NXtMt6lnpvO1h3y/g103P0DKfnouni4
W8zXdy6KN/Nlsfxx41JxnX5uCKn1vdRiaKlv1omWhxadZDb42/djanCa2eD5Uh+hwZ/MV4RuNH5z
cY4HmoVZ/HlViqO4XrwpPED/Mu0E1al2L7LKGkxWJTxeZNXwGEBt2pSpoPlWKLqGB1K48susehuA
0PPBEBqQLM358811co3Xm0A9e//5bL6EqdZvona1klvE+NPnchO5+7cfb1rj5LbfIWKFUitd09Bf
8gYYH0bjjsxYeTWLHjFjrRO+/u/iuh56aTPpL+nEwqS437T68GEIvXrlwHSux7Rrb2kYpY6523Qo
iQfcfxrSUtoSauUZSeOO1EBiD1qgrqVXwCYS93N9fp8uIbSL7OcqH8QlZGCNrIYSVd3t5RJuU3Kk
1L1UTKRqKm3wsRurmCg2PupezKgRtDJW1r+oSaXkieMXreVt8l9+fwv5Acy3sRDzAZfsGfnnSot6
phWqcdX+JFFkV0BZTURKXXiWVM779OpFX8ulJuC+lMgQLhD0CQOXOlpEcQSniSkZVE8qZ7w1T3wt
/eionHeLnCBw9FTOe4wm6dDNYc00mANROWerddSx0CoXMkOFk+ZCdpLvkLmQ2X3e0vPM7O68XMgM
IZ8LlbPcqUrlsTw3Kud9KvUKcwVOmGaUIGYpRtwTgTSWHHlrKOPRe+14PwiVu/Cokn6kVM772r0p
r/q4VM4aq5IJUSLpfHo2lMilj4ZEiw21pqTlUamcuzTZc6By7qLXM6Jy7qLeEamcM818L5Vzhpr9
qJwzJ9MDUjlnq9hyuZGpWR6V8z4h1xXbmcK79pxhgTi3oWmDv76Ts0PeWbZ9KST5qYmL2zAv0Ky+
sEhXkgqgVvrKfFZOYc1RhPh+Wq15E9KW8QZ6/yqR2SyKz4t//fG/q9TC6woqP/vjD8X0zWx+E68W
t4t3cRbg0dXXPyfFzzfTZbzyFvC1YpWJn+N16sqVLRPjCfz4/OaqjtAwcAXepgkW3lJcvL64sL9u
vopJjHF0zPKSWO88xY/+cdbmaWd+TW+YVO+5+fXyT9XH4ofiQ1G8eTt39i1ov57frtKEDepAc0zn
oNJ0Ea9q1dIDSdGKiOkKJviHKzD7Xa2SckBr+NmUXXP7Lv3sDVyczq6uARk/LwS+vxDsL6mJ09MX
0B+z+Y/RJhqq1Dm3AOGLwlWXF7dlOb2DPxdpHWTBLoqLX9M36q5Y/DSFJgj+7W0FV2heJPKaSerB
iV0ubyaL6f/GSUV0VSUcLK+qC6u/q76o7q7+XNo3i8n7N1fXwdYPrv6ufuDt+9VT7zb+SG/Kylfo
HIdTE6XPDBlN7aLHuBckpTa0rkkqepeN3qrkWMPrSd7GEyJ4GHnBS4HpdlEzuNVLIei1u+UsvL4j
7PWd5397faelhE/49V2M8AH+b/x/pluv76x5fYfhhk03+Q+v7wL8gdlXf0LfwkUKT7r0zS/hrvqf
13fOv17C53CzOZ0SGI4b/y7+4IU1LnKONCsV4iW3yHHukWdOai2pYNLvbrIEx8PuN5M/CCepcYYg
rAlBPEaCTMARBSIMdiqNaNNOKmJ4k1iX/5zf/BRv6gupE1d99DVMsrWU1cqnyBForQYMyfJ25qv1
BFxKP5B8JXwHC1NOOP6iiga8KD6pHwMpFpcAD8spYNnc/Tttk1SPq1IJ6C2jjNFfTAqm5eTU5vLp
UD3TPaDpg2aOKlQKzWDJmWZ1WQrkgvZGCB+VbBeZGEChpi1EwuW+lMKVqd3Hkh5/nhTv7C9pwfri
1FNEXgttj+ZsaSBJ+jXQi1UWZurnOpsyK5kyhTnqVM3MlU+K1RU5i4lPU/AhLbwKCa0wkGEdxa14
+36V0Hr2LcbmW1jwP1dfmiVHYm1e11nW+1nWiGh3Zp7quozjR0UUkJRWnZQ+RPIvUZLJmiig6cxw
xtl3aiYkkRmP8HDl+IgCUmOpY53d200UwCjl9cm93pl6BgDmnPnW1C5GjI8nAEsGA4BUfS+H6HxY
LJ/WNQeAns7D1H9G1I51EMN9aStNcKx0CtDPawEQyAhypYe3MB2YU7EEH7eVx/BY8C3r3kpu0X0X
8x/VivY7gPtPFmG1gZlFQzYp2m54Tq/frL7A0yqp+llKwbniSiDLMEFRJtvzkeiKMxtW26svMCGi
hp9DpeSJFh7WINaXGClqPQ3aBqfdky3Vju3XLivpof0a05KWN2C5cLMmEzZUaKWUrKiD36XUoNmy
4e6HNjbbmYIw0143s5GOqM1RR+CuJKR2wh8z/aivZAMmHnXp4ZZ7gJmd25hytCGe3iOe7GOAXgcH
0x/4RkRhxBXMcNrBlGi9wFory4SKXSFA757C5LAQkFnVoDsEGCsATrhHjMO0zqkm4DDiCN3pSseC
c86IRxAQU1mQkmMUaIIAWZJUTFkhK0N01GrHotoFAW3a72AQAMtqQqngijZNmht3P3Sy2bYQkGmv
2yDgwNocdQRmQkCG8CeCgE6SHQYCsnu4JQRkdu4+CNiRZFGJp/oYIJYYR6sjityAjLA8RTCXJdYm
HTxxGDtHu0LA9lzGtdzPCgIIzNvMMoeiw9BA2nJkhDOIOM8oJzry8jEEeGoDx4GDey+SX+IJslQY
BF+SQmApvRO7IKBN+x0MAjg13EhFTeO6eePuh0422xYCMu11GwQcWJujjsBMCMgQ/kQQ0Emyw0BA
dg+3hIDMzt0HAfuclF7nwmDZ4xwvS+SIIDARVtNUoABTmppInKC0mbuhfyCox4mwk0AAXGK+DAp5
mjw7IwDMYcWIVJA2Sh4DDfERBPDSlt6rgJgD4OCCaaRL6VFZlh4LHBQzT3PrO7bfwSCAaEO5NEZy
3TRrbt5u5VrrriCQabHbQODg+hx1FA4UDNKjDQY1SnaaYJDuCAOZnds3GGR6RSM1YTiVMuVGwWSo
IkNWMYuoZDgEkULp6kDBIPPMYMArYQMlASnF4AuRw6u44Ugoj00pNRiKegQDQTiuATgAARx8gdCA
TAxpGcCsI6Uy8JoewSBzHE8AlsdEC1xX3nyydn6428q5Np33A/LsdasncFhtjjoCBwoGmdEGgxol
O00wyHTdD8jr3J7BINoridQoAYtTbRBjmCFeSoecjwbJkgurNBOsqmZ9gGBQkvtZQYB2VCsaPPSk
t+DYMQZNVQpkbBmxdF57xh9BAI28jFFJ5EUEHFBWIxc9Rp5bGaMGd6CaiToGgx6338EgQKaqwNB+
GDdNmht32zjXDzbbGgLy7HUbBBxYm6OOwGGCQUn4cQaDmiU7STDooYfbQkBe5/YMBqViBz3OEQiO
nSAwm5XVtmXUyMKaFilYqRrtgtGmHUlQdjAjyf2sIIBgY0rmDVIiQUBqKkOFQBqcOq2ws0GXjyDA
BiZVyTlIRNOZosRhGQI0MOfM+iiDqHYmOwaDHrffwSBAGCzS8RvWuIm6cbeNY/1gs605ivLsdRsE
HFibo47AYQJBSfhxBoKaJTtJIOihh9sSEeV1bs9AEO3FjUWt5KIMBFFawsQJoiHDpUGOBemVIETS
dkfJsgMZlD4zCIhUlsIKg0qpwRVw8AWLfQlegMSl4o4Z/DgQ5ITmloSIIiPQ/9ylE3qWIya4Cwyb
UJe37RgIetx+h4MAgamhUjeHzzfutnGsH2y29W5Anr1uhYDDanPUEThMIIgemZeur2QnCQQ99HDb
vYC8zu0bCGJ9DNDBtBd8OpnJqEQ8Ra61YRgJmPAoiYaBE3SgQBB7ZhCgGHPYEJUSgsCps0GCA2Aw
klE5EqUJMT4+GxA1c1EAXBDrwG0ALyIR4HMkVWQ8ZaFHZXsEgtiI9wKybbYtBGTa67B7AeMcgQMF
gthoA0GNkp0mEMQ6QkBm5/YNBPU6HsYJByAyApXV+QViBbLgu6BIbBmlta7EnbeD9wQyepBcnsYL
8LBE1KpE0SrwAuDnkJEEZnQsDDxtQ9CPIQAaD4yHUkRkSgwmWiJnU01d4+EVpuRBkR6BIH4cLwBa
TwHy0cbo+cbdVo417xwIyrPXrV7AYbU56ggcKBB05NpJfSU7TSCoay2kzM7tGwjqdT7RUm2Miw4Z
C//hpeTI0lKgELxP0aqyPNTxMDrwCeGDQwCXDlsGiE6wSIhuS/DnMEE8BMeJkV7Ex4mhQkQmlKPg
BVrADF1apDU4hSIQZoOTTsnQIxB0nBPClHDFNduSR7lxt5Vj3fmEcKa9boOAA2tz1BE4UCDodCeE
O0l2mkBQ1xPCmZ3bNxDUywAluCVYptNLIWUvBl4izVPiahTYaC9wDORAgaBnd0KYeRKcgmV/mdKC
tIDulJyjYKX3ilvnwmMvIBFyE2hBJEpZpsMEHpoVVgS+tKEsq9CR7xEIOs4J4W6T5j6b7XxCONNe
h4WAbG2OOgIHCgSd7oRwJ8lOEwjqekI4s3P3QcCOyqf9DdAZsPgyYiS0VynZ3SGtOEUehqYLNhLK
O0OA2D2FPbcTwkbREuPSoCASBDBskRMkIOc4ZoyVMpDH28E07fY4wHzFIrgNijFkHSWAGd4oibmn
YidJRJv2GxkE7LPZzieEM+11WAjI1uaoIzATAjKEPxEEdJLsMBCQ3cNt9wLyOrcRAtYcjXyHbGSC
afsSIX/7bQUQ+3ZVA6TtO8+lPz5Get6Pq/THekyQ3WOCd+FKPTyx7PX0zY31MMY/A3uTwqSNKEEt
eKQOI6MdQ4E772FdYwijg7LLsh21FFOLpbITI2yxePcO5uHPPI6lJR6Wa1yks5xKp2Q+jGCVp6Im
jmgShiXj3WNgCfmOQsbL+T0ZL/z5WzJeSgXGuqbj3cHDvK/zNTuWNvhBG/yUWpgYmKFrbfpWVge1
THMplY+bXJhRmB2bqiGcmFxYGKprRnGietf9qZRsYpY+IrnwkwpubIe8bCL7hQ6olIGVjAKKBMBR
zsG9pRa8XfBvAU/LwLXvVy6VbXeAV9J3CiDAC+dJk8Une6qe7nl/Vwf8/v1NzndildmP01UNbj8P
yW3FyZP1y1v7Fj58D84ufH4PwzwV+3tRsOpuXVrx4X5I82lyeTem9LTmfAszxJfJpip3+DJlq19U
5rb66odJKt6bQVOfLyDOEDARwD8Rj1/yRvGyAkGDtp+RjAn5REBCL2WjhFmU1kNKWNdOeCKguCSN
8uUsfQbu4AqGsyywoSr0vmmuc7Qpc4rbiDadh+95+J6H74GG71FXKbvCq11UOGaQdRj5Bgy1du/z
tudv8rq7MeDaJOQ///zt1y+/3iqmSZZz/fKvCVrg6VffvKp91mSCt7CKhwvFbL4s7HtYn6fo4mWH
tyRXevWWtFd5/5KvFtfVMrLhNZPiq+9eVVa7KH6epvBDLK6nVRXOy3ZuDZ9gca5l29QuJBXAHptP
a6SCSb/yafGl7OvTblNypLVsmZxUWzlnY33SLlSP0FhhOceFqYyVXOq+xpqUVM/IWPWEmn6d8vs0
Vj3hZITRQmUI4H1lrPTS9DXWbUoe0VgzCkAmMdMq8KOqeglKC8xHUvUSG4alqcwOHOZuZS+ZmZBO
m0cfY9nL1FjGjKLsJVWEGFbvTdGeEw5f8dqc0eZpu4gRLo0IIZpU63hDeq/jKyVPvDTK2OTmXbeF
P9osCmgxgs9ZFLlZFKm50v0RZFEwIaih1fiWGlC/QxZFpY0YRRYFNXgwoKIT3jOw8vsEKjoRjXP4
qd0iYjSvA058iM5vVvKUSRQ7jockeWVuLuzT5i+KdeC3WCVkFtNFcXM7myVp3O1ynZMaLotvQSN7
s7x/0MVyfhOLOIP76enp8j8et3uTHluyGUAN2YdgdKX0f83nb1/OynnjmYIp3Kg2nVIOZtoc+npe
/CUu/M20ysFI+0TgDKS03ukd3L69OKQzMb0GcRefkdU/1PCf9b8k2DtQrEoTvl0kyf18NgPVY6g2
vN7HG7io/p42xeZgQbNFJX7OOuHFn/1y+j5OcmReP5uzPbp+NsefXj+bA9SrZ5Pa9Sx4lfJ3q63I
vMM3G9uKF9Xnxbvrq+oXFHxIyczpzsvvvvzu5cZL6hamH+oWXhtSi532+667vXjQAMz7p5QpXLGB
C6EV03i1T71piFUqtn0bb5bpB/71w+qby/my2i69gK9xQwgWVD9WkLXbbN8rI1fCUCUV13rUg2VH
WwmaWNelNJw/bivcbud/f1sxZQTlXML/2nYoBUCTFEtM4a1NXZq9/b9XTFCKUpA0ZSK2FZNInvjr
ldZc4yYxs7MA9otpmJACY9NBSiyE4AYzzPFvu7wpFWA31sruTNIBbEczShCzFCPuiUAaS468NZTx
6L12v6URPSPWGbHOiHVGrDNinRGrO2L1oorORK2eyWsZKhwvee32Yv2rWQf0wZsu1q//NpZFu29P
wN/+RwoEQMen7MSLYu7+XSH9LF580a23j5tH10UgmcqcbRHoodfWN8AWb35JlrScF6tb9/3VsrkH
Ep/pVuJ/W18IXaUuPsGFraaZdHT2ZtFwUKSTGpy3VKOb9NNFUR2ITRNAOrObAnvFtCzs7Bew8nQ7
HRGu5o/5TTFdNmQodlJP0FbqweK6QZyOXTYpvoSf/3/2rq3HjVtJ/xXhvDgBxDHvFwN5ODjZXWSR
nD04AfK0gEM22Y6xHtvwjJ3sWeS/b7EleTRWq5vNvqjl4VMcyzOqKhbrYxWrPjbxb1/OzNMqqwU1
MWintaAmCPnyZRyoevkSIIaBoIdf8Z/w33++r34OHz41Fn95F/+0r41+eyjRPTuq3d28Cvexfgc+
83bjPrz2r8Lm99f3v5VcqORCJRcquVDJhUoudJwLDbrZ41shx11ffp3Xunwr48Pza7vWpQZD/I3X
ukyMno1vlDTX05rN5W4CqTjrqV1MG8fBhZ2VYzgCNM5KxfgehKgkW0ezXEeXMo9dyuKJtWZHpePM
3SpaswkEyF2EHN+gqbcKj9tYVxNwTlLcDu49sIse9+S2VQZ7HRiCk3dMxqWDM7IkqGKOSMVopYgY
R1JynqXzIP28JCVDvv+aSUp2HZMnM+bqRq2E5WDXrn8ioDnDIrAangOqbvRaeA4gop7Ih2/YSmgO
mjbXJA9sqdT3hbns1obEEHdhkpKyfcv2fRLbd9FTyqB73gQVLkpSkiXfnJeryWs+9G2YtOVOuyES
XfQhjZh8ApKS/m8RlyMpge83sq2+dNm6BOXKHEZ3x3KLnlXxAlWJjulZbrY0JrIrHEhb36hxNBZd
itS3e9SYcdqQ4BrFWobRhjiqwFujxu3FqyllDLIL2WK1wtqpbK4l9/NbYzlYGiVXUjvtmAMVuZOT
T3bQOFqMl0Hj1EHjaC6xFIlEz6AxU5ST3WwuzqNrj9rIHFCfYdBYcIF306ajbyUF22L9RIruA+1C
9Dga+3kmjVUsIjVrz6ZY/HYlLzlp3HHzBfJSyh41++H0Zr+/2TfNkPCX3X6g1y4JPG35+/hsb5rY
PvfD902DSVLvZUse3afXqIfUHQ1ea2mR5hSEUnWNLDEEYYk5FsIxb+txNzzq7A3LQfrcd+w+2/+b
oc/S9VwKdYrMYpto9qz3QeTWKe/YEXh3/2z/xx8OvYF3n/3sBfjgP37aNw/+eOi+bCu4dTtNo0Pm
E3JpDnM8CrigWpfZC4PqiAkqXLSOmCXfnHXE5DUf+rpc2nKn1REThJy007wDe4ZtslNVuk7mbKu0
nhpH27vmp4TQPpXMmLDBINN0TAbEKw6pZqAKGeIDqiofJNYuYDWySYJ04VEjfTaEPqxEXIWpgTTB
7iexpOEPPBdL9r4UV2Lvgs1oz9/tbfju1yFDLr/CbwJFf2W6Mk4RjlhlPOJMGuQUNSgwGYucwgfm
YfH237v53d7B6dPHtds82z22+BuA4/NP/u72+d5fn/92d3vz/n+fbcDZw+YZlZw+29j7zbNTe7fM
dOWYDNK67mmhV69jLtQ+k7b7LHu8K84ZNSlUnGCA2BLN4YfrFdl/TqegcM+Q15d6jZzyuoF87/fD
65AbWOqDOpBRHQ+xDff0VvUoS/L0Vx/s2/tl3Pw/4lc1s37Nl2epuQukOjYidgTSgzH+ur91HhhM
X34+wxx+wYvNELtsvtmtbN5KZh03EjWLxw3Se9zoRASNx2QojyNUYaMq82xlnq3Ms5V5tjLPNpDb
oxtIG5DKK0ElImlhoyqIVRCrIFZBrIJYEyPWomXCQbcLfSoseruwbjaqLFPNedGRI9CK2KiyxF8f
G1WWGtfDRpWl3urZqPq0EvMX7frvCBOELGxUJRcquVDJhUouVHKhRXMh4SQ1Lna/aPAkHgJBxuMA
MgqDnYoAbTrhy0AQTXvJ5d//+sOP//b9Q3frXyZ/yGWz+Xv4fXP3+l9h8w1Xm/DHPezEu283t80V
2B38xeu7BqhP/kn82Xgr9iuASfwrv0Hot/Dm/e6e+zZ+VYx6H+A3gfFu/vvtXz530LIJbBjfCm9s
SMCpHoz44y9xHDDeeYK8P/7y7ZF6gO596h39k5uJZDzp+yX06Fj18z9+uvkp3N3FYAWnn2jGRw0b
b0H6vXVv715t4LwV+3tfbHag/WKTEn22m0/v3ny8DYBXlHhs4OgocVCIMwLY4yxFglKIndxh5vEU
arf1umt+tqX7Fv5BPPn9Lbp8Sqf7u8k63SEpiL3T747HM3Z978cN728/fff23T3ssuq4+70Z27gL
4X++k8cd8OPtp9peJUu13xz8Q4wZZXYERG3jpdNonHfBwSuvmYNcoxaaQeoRV13WAlBDVwZgLSj5
5XMbn6Hj/57d7rZeDPrfx9rHdvOZBGCSQP+gV04ZLFG3YWWwaZRasu12LoknrE9N6SkD0+5EJ2lN
uycQW4HYzWBXzKR3DTSfZT/6+28gkv9zd0qJYgMsf7x14QME3s+Hlybevxg6XSO2pJVD76mPVokt
pesjKsBwWCO8GQLWejSB4lklV8r2KdSWsnEUrF+ns4JdzLgnw2dxVibAQo2zUj1+Yj0q2XYwXrGz
mrbN9fUd5Me7bzO0suSRPSHSKi5VwxoqdCttaAILqtBbEee1nxT1a1Sar4b6FQCO7Llf2yaRE8hf
hNkN9K6QUWF9TDnRWFn0EzMw5WgmSDOFrlsfbh8UogyEqPLWeZtdOBlnl1mCt9CU7blyVGfwPntP
yl7gXDKNJ8o9s7OYLtwzadwzjbkoXopVrId7hsApes8901ahTFt8ipeK/D3cM4aY3ZMYZGSO3KjF
aYn8bXYRtO165tLcM1QKzfYsaeO4HEFJtju/l8U/tYtZ4Vs4XALyN2GM0tadf9IndfYoC0ryLRlH
FiyVDFLXNRIEcJkbz5Fh0iKBveQAOdLXeNy0Pj0zm/kg/bxPGvR8P/1qnjRo8PGU0fvGrIQTfZfs
nAjIzwi4Gk50zm7ISjjRm0rBiYDiDKv8JUjR46kmyQdbGr77Ah3NvUBODHKXftSgbOCygZ/GBl70
pDJoYChBhYvSkWXJN+eUTvKaD+x5SFzutFEDev7Bgb2YbIJnDYZ8yyTPGqS/MbH7fsGvlPMzQa/8
wFJZ4h3THGGnAOkt9chUhCHI01xlKuGND/O86vYg/eo4PztFplHI1XN+djtNo0PecTLRYWbi/ExW
a9G9MM3bQQ8qrPPtoC75LvJ20JdrPhBkE5c7+e2gTiHN9XB+sk5VFBXXx/nZp9KoLmZCnWLeOFRT
q+CwxivkHCVIe6G1JHVNqB4HoawLjxrpV8r52Wd3dXHOT/hfa5S3yOpQxZM2ZJmUcaSd5NgriUOQ
i3J+5pjsGjg/c/S6Is7PHPUW5PxMdPNezs8ENcdxfiYG0xk5P5NVHHjcSNQsjfOzGxFiOCicn4U1
oLAGFNaAwhpQWAMuw6DWA6Rc5JagEpG0cH4WxCqIVRCrIFZBrIkRa9Ey4aDbhQQVCudnmqnm5lSY
RKAVcX5mib8+zs8sNa6H8zNLvdVzfiZoNXfRrv+OMEHIwvlZcqGSC5VcqORCJRdaNBcaNF0mthIX
mp9Wu8g2npRL0/xQs2dKYe3Tw4OUVFuIfE9j8Q8z4rzTHI3iSWy3p6s7Od1tG4dCqvicHK3gf70P
Hxpi258+3oc/dpSzb95Z/+nVHaS1hw/jqet2s/tk8+7wt88OyaXf3P8WHv5+cxt/V7JccjFinhS8
G8k3MHCPKaHWF0gEYBzTe9YO1cHa0bOyMmtlZ2AfAiCF82bDQdM+jN5LpwQK6a2ILRZPiENqp3Tk
j1jFKhIqDB7DIQX6mF0j8QqZUdbGIbUzllqKF6WHQ0rCebjZv5D5jeKQatRqItNTONkMtIvWK2Sv
hBVnfMck0k4jk0DxQ3JJcZ4sh1S0WOw4X6HF1sghFc0Vhz9WwCFFY6jkO9alDpjsW3yRQ7k2PYcU
Y9LseYSa0DAm8oNa2hQaoTa7wPl4fZGfMwj9OxohNpK3GJRkWyPHKfl1Lj7fYtrG53zpahYD/GkW
n+HWxT+5d5MdSoqtjhT/VzkYnqBXfrsKNYQTLTi4r3UgFDZIB48Rs8QJxoOGI8i4qTZ5doahkV6v
cDC8U+SmjWb1g+HdTmM4z+3KTXSYmQbDk9VadC8Mat1KUOGig+FZ8s3ZL5W85gObPhKXO63pI0HI
lQ6GDzosyC1m42oEX+chSm4hT1ndIYoRSbXeVUzH0nCCikLq6+ME6FPJjEEM4aR1gWMQioJQmCuk
KbXI10JVzKg6GD8Xs+hB+pVyAiTY/cKcAFpUFfG1RA4STcS9rpHVhsNPGfgVQgpi3aKcADkmuwZO
gB694h3DNXMC5Ki3ICdAopv3cgIkqDmOEyAxmM7ICZCs4sCTZqJmaZwAnYggMSmcAKWruHQVl67i
r6KrWGqlFeZy18xauorX2VU8EEgbkMqrPiYiaeEEKIhVEKsgVkGsglgTI9aiZcKpaP0PKhROgBGm
utALA2cFWhEnQJb46+MEyFLjejgBstRbPSdAcuSer2iX9DhHp5C0cAKUXKjkQiUXKrlQyYUWzoUS
pjlJnOY0X9OcbqLShq9k+JEa+BdqNwfBx7bC6y1ThdihzS585DPT87TCE8GUbNq46I3IG18mcZI3
y5mveHw5Kh0Hflaxg7GBQwpmuyH0ti2cMJJL4kguW+VU3vrml6OxjFrH/DKVRtDzU2xD3pkCtRgZ
V45kFutKcWQpBGKOLYNDCZPIMY4dZjzUis31GOBB+nnfQx/y/df8Hvpu1rPJgI+fstVn3gK+wHPK
jeOfCAiftQq4+HPKu2PViXzkRrTKt/hjynvQaDGgahVw+deUdwO6SS447NG/x3FucKNAWoy78HPo
Zf+W/fs09u+i55SpHmo9qLDWh1rPy3ehh1obgXJfQ09c7uSHWs+/U96IOcVr6EO+ZZLX0IeUMyje
clbKPG12aaLRyso8hGlGm2E91Vke6GChoZG3ZZ3lgZXSHEWLKbFKi62R5iiay+Tw581Ac6So4GQU
zRFow/A6aI6oMISa89oMim9020zPlbh/ahexQt5mGuN+s/ZMjuZtpnwrJSuL32YXs0KuXQJBZ1cZ
1mr8zpdbhccp+XUuPthFrpLjSkjG9xxXbRd7JwlHx3U1KKljMniVHFd9etFx7dfMWy0rhEkFQnFL
QagavJmQilMbfGBh3H0HP3vfcJB+dRxX3SLHULl6jqsepzEqf8osyWFm4rhKVmvRvTCoppagwkVr
an3y6aVragkCZTYxJy13Wk0tQciVclwNoeuiaivUFRId9ak0iujIGUwYYQpVqpaIE+2QIR7cyhMe
uPC0dnYumsiD9CslOkqw+4WJjmovqFScoNpJhbizFfxUTRDsfwEy19KRelGioxyTXQPRUY5eV0R0
1KOevjDRUaKb9xIdJag5jugoMZjOSHSUrOLA40aiZmlER52IIAkvREdlVKqMSpVRqTIqVUalLkUb
0Q2kMr60mFeCSkTSQnRUEKsgVkGsglgFsSZGrEXLhFO9oHFQoRAdjTDVhR7zOCvQioiOssRfH9FR
lhrXQ3TUp568SqKjBK3mLtqNegfnIGQhOiq5UMmFSi5UcqGSC62N6IhGzh/9xIiOQGkdgXkVNCmE
SHV4syyT64ZG2hf6xLhuQGkZ239XsYhYGWJ28yyi9eW5Qd3bGo4s5dHBNrtoOm6+c56udqXiwFZ8
tlu1zrMkcPdQs2UYr3Iub31ER9FYNGfsc3qiI46p0btxFiohLHy59O/Dh9fv/OvqOVEdmSLDWzDH
mEKkqQBARRUQICwcR4LESGMdkLIgoYb/h9PaoH7Fx4Kf6UuJcsOfsjsVf3n35uNt+Pn1v8I3d37f
pJh0/ttuhjY1vr59tf8BTmzlml9LKUMcDv7IMkxQkODRpApwcos/8Ondm/0PMCGChl+HaslBjgCH
FlvVGClqK+q19U67k7bJTPsN6+t5sF9rS8/9h4/hDj7cnTINhSxMKdmcKd/Hu9e39y2f/jnAZwnO
ve5N9Nfj694FtVl0B3ZdBQwTfskRgwGSzU3YMUyUrJJh4uK2lgyPxNOp4uU4IFPcOUMhreIxcHpi
kBNEICxkxaqa2VqKXAjQ3SEsl9/uDAQklrvyIcBYyJprXiHGIaxzqgkyDgeEmasd8w7sKB5BQAiV
VzWH9J5GCJA1QY5Lhaz0wVGrHQuqCwKG2G82CGAAb5QKrmhb0Dz69M8hPktzISDRX89BwMzaLLoD
EyEgQfgLQUCWZPNAQPIKD4SAxMXtgwAzpwNiRYOxAFMVtRDXJIWTamAODtBUWV1VWNlhz7M/Frwr
hI0Y+b0IBBCI28yCbYLDgOjacmSEM4i4ilFOdOD1YwgAk3qOPYe0WcS8pCLIUmEQ/JAUAktZOdEF
AT32Y4tAAKeGG6moaT03H3365xCfZbkQkOiv5yBgZm0W3YGJEJAg/IUgIEuyeSAgeYUHQkDi4vZB
QF+Swsc4YOVxRY3wEKFklLE2SOvao1CzykpJibPZWUBPIYNfGQTAXwFwewXLqRTiRtQI8gKFlJc2
SB489eERBPDa1lWlPGIOVp0LppGuZYXquq6wwF4xczo/m2m/2SCAaDjESGMk121R8/jjQak1zwWB
RI89BwKz67PoLpyoGMRXWwxqlewyxSCeCQOJizu2GDSqMVkJ6WqsHDJcGcQZq5ExniHDagEYhnHl
5EzFIHFlMFApYT0lHikFVuLxBUTLDUdCVdjUUkMyoB7BgBeOawAOQAAHP0CoRyb42ATCrCO1MvA1
I4pBYplMAI7HRAu8a8k6OTs/fDoouc4e/0r017OZwLzaLLoDJyoGtfX8rqMY1CrZZYpBuS3EiYs7
thgkx11IOVObSqJKNnEtMDitKgihlNPa1BYOq26mYpC8MgjQjmpFfYXgdwGiW8Ygp6sFMrYOWLpK
V4w/ggAaeB2CAtuKADgA2R9yocKogiNBCBrSgSYS5RaD5CIQIGO7KNgP47agefTpoOQ6m4Qu0V/P
QcDM2iy6AycqBi3MOjdWsssUg1T2lXDS4o4tBo1yQEUMhLWaQaT0BPGKe+SMBMAilRBWChV8NgT0
FDPUlUEAwcZA/maQEhECZIAfpUIgXWnABuys1/UjCLCeSVVzDhJRhXgd64Dea8gdOLNVkF40N5O5
xSC9CAQIg4XQ8dmjtqB59OmgxFpnZwFp/noOAmbWZtEdOFEhqJXYcxWFoLkpR3NWeGgWkLa4YwtB
oxxQECyZwxqSlNrBSVVbpBWhiBBbeSm00qyeqRA0gsLyIhAQqKyFFQbVUkMq4OAHLK4gs7MSA7I7
ZvDjQpATmlviAwqMwKpzJ+IMEEdMcOcZNh7rzivhIfabDwIEpoZK3V4+P/p0UGJtciEg0V/PQsC8
2iy6AycqBLWRcK6jENQq2WUKQSYTAhIXd2QhiI7qTFYaIhOHcCmIFIhbw5BTEqJWVUPAox6iXTVP
IYhOPBswOwQoBotpiIoNQQxM5SUkAAYjGZQjQRoPad7jxlDNXBAAF8Q6SBsgi0BGG46kCozHLvSg
bH4hiC4zG5BXPU/22cFZQJq/TnsXsM4dOE0hiF5uNiBLsosUgmjubEDi4o4sBNFRswEW+5rEYoXg
0sWnRgNyWGBk4H9rxgRRYqauIHptswGhgiOiVjUKVkEWAL8OGUkgomNh4F9b7/VjCLCu5pRRwHyp
cKSTkcjZSJthKvgKU3OvSH4hiC4zGyAipQMgH22tnh99OiSxfvDZoRCQ6K9ns4B5tVl0B05TCKKX
mw3IkuwihSCaOxuQuLgjC0EP4mWloZzXHJweyQqCG5c03nGKGuGa1QECnsNWzVMIotc2GwCrhi0L
BH6jkIhXtkYuTghz7x2kirIS4XFjqBCBCeUootQCZugaskDtCRKeMOudhPOAzy8EPbbfbBBACVdc
szN9lEefDkmsafZsQKK/noOAmbVZdAdOUwiil5sNyJLsIoWghxUeWghKW9yxhaBRDsghHBGOBVIm
TjELbZDV3iCnna2YYDF4zlQIYlcGAYZVxDsFx/46tgVpwZGVAPDeyqpS3DrnH2cBhgRLBA5IQNYX
hwnibIgRqKqtr+umdFSNKAQtMxuQFzT7fDZ7MiDRX6eFgGRtFt2BExWCLjcXkCXZZQpBuXMBiYvb
BwFiTgfElnPD6xopH8deQyT1xMRCmFKVMMRCUsxyIUB0h7BrmwswitYY1wZ5ESGAYRsHvj1yjmPG
WC09eXwdTAmR1NWw/ixA2qAYQ9ZRAphRGSUxr6joJIkYYr+VQUCfz2bPBST667QQkKzNojswEQIS
hL8QBGRJNg8EJK/w0AnhtMVthYAhVIggJPhtoYhssYuJTKgro4gkjAnesASqG9VBEEk6NCNbhskq
CSJvX7/6YCvAtOcplOGTskQy3G2x2Ia8QouFP97D1n+ewiE9Lalmj4OxHAbSHFJNzj+TasIfvyTV
pIJrruJ2ITfs/HbpW3y2FEUoxw/a4BNtGAN45Ttt8EhqYMa2YJ4S99vsIs3q4j7mAmvc0EJT2hr5
T94T6TqZ8K3Co0YCVWD/396xNrdtI7/nV+D6RU5HlAES4MNz6UzOvbZuaufG6Vw/VDmFD9DRxJI8
ouQo0/a/3y4oynJFSeBDMpNQM04kPncXi31hsRtxL8IaVrhtzQyFEYRBZAjX9ySPQtNjxXaF54G/
zYdZQl/KB4QXThCT5GRPc/oi7y/gQ63en+c/FWiVEU4i9DwoOiPhbK6q+GNfq8cNBtTZtAP2w/kI
5RN6LWsiUnVIgRl3jpysyt710tYP2DJreWuxPhlaAFINANPCyBsAii0AajenqIuCadn9DfjsnpcL
n3ZXitpGOK0onwMgywVQux9FbSOc6jUtFszpi6Yt5wpnD+rJuLWIQTt/2/nbzt9Dzd+j2imF2mpq
oHDMSFk98B2yl6X2mBfNoNUbbr2GfADkby+vry6utoLpIueMLn5A3QJwXr6+TL3AN6pT03MCB8h4
MiP+PTgGfnArc3oTFnkLpp2tXvIqGSlDMuc1XfLqzaXi2oR8HKJDL8lomCSAbK+YNyS61KnWKOXL
9BJFl2HVr6Z5iQxsHK5aQHkstwdULUja9np/JDj6L3zi3Qi/rp56MngPjs9t1oYS2BTeOVi/hFym
DyQ2UToCZCxgoaiEsxJfV9Og0HrgBScOJGyS9hpVV2FZwMVsHPUXzOovQv5jf+HaNvyi/YWU8AP+
vPAnPNVf+F5/QeGEjyf52/4igi/UevWtcQ0HTbgywDvP4azzrr8Iwv4MfkdTuvZhgbv+s/PMjH0e
8dA1HLA8DS59ZrhR5BiOZTEhwQC1rXA3yVQyqIqDYWfQ1JldSdy14ycwecGdHQP6KGyfk6v5KJBT
MonJND1M1BQ/I+yZLeKQhY5vMFXiWtLQ8Gw/NsIwtoLYpqHSfkWgAr7OA6v322T6QU7TAziIyzG6
kotZCqVSFEQHoAwNmJLxfBxidAAP4QOID8JqwaTPGaffqVapZ+QkvQygSHpgK8yG/i02lcZoqrrc
iR2sEuA5nud+1yWWK7pPzS7P6xiZSpljsRXGpkvR//BxXYuDhnY5ggNmK3gfVuwX20JYD0IbMoNx
+wE3kAS9S+Aq+J6A5p7JcaTcRcVqqz67j393yZ3/6XYCXPnUIkKPQtuCXVsJZLNqBDojUl2I43yv
0gqIY7KIeqDobSodg1sMvOfANw1hmuCe8oBaEcWOxWn/XqLjKpITbApMdIyJ59j3HFf0iQNOax2M
hbl29yM469FtVkE4ioqsEyXzaEKMcXogwSPwAnJ7n5IS75uM4+EN6ZBI3g+VP4BzJQYbQ0YDbOmc
gB3w+zf/U4s9IxWlPP3mLRnejCdTOUjmyR08By5d3g7W0sfpcCYHoQ+KWfVWli8oiYYJ2pkDP8a+
v/DwyXSQWlcWuEm3cBDfQjr9Tsf/c/1Vlg3jJQPL5zE2zwpN+ujDrT/xnq66c/pn71v1k7wlfxFy
czsJQML+QQgac8A1A+xDDAACgsMJADlM5CAFFi9A0FWD8UEMwK2OzOEiIBmugQEe8NjADz/M7/Cx
Uzg4HA9GwFkviKCrA5H/CYmGV3eAwj6Y7cubxvBzxTAjLX481eLxsmapkzJda65v0sVuYL9HkwrX
Ymiucy93UacwknkrV0c01zVa5gKYLm7W/ar6BCPSnDekxSxwHXcV15m5y8gbIYodvXMBMQ9XzwcD
tM8GAzAKKWjF7BE/w//Xd+EbOb1X8aJBgt9WBsG5f3uLvNNZUkJZ9Bg/uRulIZQOavtgOoxupDK6
McS+5G9U6xffd7AbtlZCZk4Uah9elbZpBjyMHOZLwwljAEpQgWn8rmHZjoht2zOZ71dbIXW2G21L
6Mtmya7of1I06XXPoupukL0KRSpWIOfmpSZ3I7BdOsuvFxiOdsDISFZ8Bo48yLtOeuwXYFa84pe8
cPUepvFKF6vQZJj1BNWnQOuoc6FQFF4DhSeNwu+D79DdLcsAVCoKrzncelF4DSDXlI+lr3yWYqKA
7ik2yTZR2ZUz53YZZ3Xr0YtxPDmsCt2HUrUmdK4jnNjkhgSTxuAO7oSzRAB8JCkNXIe7tKIKZdv1
0RL60ir0YSRwFOpWpBp035AlzPO2y5IsKAMjsWRBnCHJlT+SL95lN+nA/g6eBIi+i0MndHwZGzbH
emZRGBmB8FRnVOlK5vihg3uYlu8FTz0BFyLCsSOdNLrxHpTj6X2UjE6X/Hr6Phn17j51CDC7JB3T
5mYHQ62dTXp36iGZ4GIryZRiuBmiQ5udAcadflKB8AnJzq2Uwmp2apGxQ+LJVEU6CIY5QLYgOaKa
8LIpL4bXdUkkhgnMSyl74Cp9zOI2BIY6QwfcYuKHs+G9xMjMtCZONy0tTr+Z+uPZcdj8R3yVjDJ9
XEGQuqazU5BmxHi5TNooKEwHKxsme8AZKUIXcpKObImRLN1MTw8zNDfYXnNjt0YQFXZP/01C5fop
QxRd+AUjtWhYXE3I9zIJp0OlDtDGGCYTDOgOF0pvF98xAPcnpzpZYKfDEYbzT9nyY+T8k30QMLQm
VIA4NYnCyXgMqMtI5QVlVhHmDqngfaLA10mXO3up5ENXB+bsWp2oT3atTkJXdq1ObtXyWkQ7DXcO
5nOVAKW70XQt+6pjLc3KwTCzO1Eb4JmLN+dvLtZeklLY/CulcMZIBTISV0M37zxgAOz9AQW4SjgT
wnUsl5ppPcR1RlQhe/9WTmf4gN/fLu+cTWYqq6wDt3GPMSpM9zGCVrGkxL0wwuz3TMeGOe82erLs
oJUwscMIuEmcP6YVLZYfuZ9WWL3MdSi3OS06oCbzXNukNjXhIXlDqp0muRdMQMo0HQ+8SM8sCiaz
OfZqcVwXlEAemNrJkvvB9CxhC0q9ElBSIQT3qEU5/fuQ58Vq9ihSUX6PtJ4mXQ9BtRqr1Vitxmo1
VquxWo1Vg8Y6apiw0OqCBgrHW12YFwvDnJF/riI41+AGF7u7S4bJf3HFHwYeN3F0MMtRafqx7HxX
brSPu9BRBiCbbY8hPYxafuRPnSob+KsJfMstBP51eiAqHa48oeuBvKR4/CcfDb4nULmBRqU4ZRcF
gMpInMxnZBgTf/wJuBxPYyqZkh+TKRnOcjZylEJPmIXQA+M6B5ySQ9Yl5/D4aZrOrvJHymF16KDd
/jXCfUDaNa8R5q+rtb5Q6wu1vlDrC7W+UOsLPfhCWXborjwWt2sz82gpsTqDW1NK7B6kMfu6ESmx
1BPctVQidm6dqEJ52F6XtXtmc+liNnLPLExcR+2Zddzqe2a3IfkESfjmTjAt025kRTt9pV1rPbs9
xOJliFWmApxlrSrAwdfNenae5+yoZ1eEVTnrmu1moXy6iAZWgGNgzVhKTrl25c1CiKRNmyGndnjv
HIsv8kbKqYbWKkWKiWZWd21irVIkly0aUauUm5agSra7zAQLtUStUoWN24hapaYHzttSU7Gqwsrq
mk6esPraNZXVtahooqYS8Kc42atcqFYhmVegvKFVaLjoWqzaoHyZzAp0Ec0rqK6Y1fbqYlZEMq9a
RlOZ1QGFwVpmzaOL5zSRWbEUZE31vRSSeYGqZhUM4Lh33v3KCgYg0mi8NSI6CqgL00mjo1oFA3YE
U7jb5biZ5bMsGLAPr0qVsWw7iMIInMcwoKbBaWwbnoyY4Tiuy3jscRbIarsdza17WzLoG1cwYDfI
rvcZFAzYwzSIQ7lsbU2GOVDBAG20jjoXCqX07UHhyQsGlILvkHl0GgCVSgbSHG69ZCANIBtaMEBj
1ZTjqqn9lS0VI9LopTTEGHK4pUxwq+dW7MMCiHmO/flVfdiHklNF9rvcj4HJpGF5oYMVbS3DF25o
+KYUHngCpsP4oVrLZNA3tOqDBt2fuOoDDePApSArYszo5EHkG35sC8NmPhc2Y04Q06NWfShDss+h
6kMZvD6jqg9l0Dti1QdNNt9b9WEfmm7Vqg+awvSAVR+0USxoM2piplf1YYdG8LqUVmiY3FZ9aPPG
27zxNm+8zRtv88ar7aHdpUiXSqpcHFFTk7ZVH1qN1WqsVmO1GqvVWPVorEqdHUuHCevq7Jih0FZ9
qECqJ2oyuRWgBlV9KAV+86o+lELj86n6UAq9xld90JbchwvaafVn3QdkW/Wh9YVaX6j1hVpfqPWF
nqTqw65kJNxW7DVyp17z9mAjsWynEXuwLdMDRNKdbZX3i3hdTqvtM/gy918AXay8fTRPvf+Cgp5S
2y9E9V2NiOMTl4rYsKl3dLQFeD2nUlZvzG1TeJEJfGYhx9k2SBSY+FTQUAZ+bLqiYob79j7GGfSl
MrvghRPEJDnZk6C1+/1u2f52q/fnpQ8UMBfDSYR6j6JqC2dzpckwtvNYyaqzaRbIw/kIxSkqvrW9
yspLAAFxjjyl9GnPVroQ2W15azFbUQtAqgFgKqQ3AKRbANQ20OqiYLqlZgM+s8dy4dO2zOqDD7Nc
N+BjPSsXPm2TrLYBTveXa3FgTmhwj5hzSzdx1BRxa2up7fRtp287fQ80fY9qpRRaWNJA4Un3HpWC
75CrOdpjXjAkrTnceiFpePNvL6+vLq62gsmQc0YXP6BqgasvX1+mLmu6Neg5gQNkPJkR/x7scz+4
lTnR+f1vMVdvwe0lq5e8StQWpLzXdMmrN5eKaxPycYj+uiSjYZIAsr1ntohDFjq+wTjKL0lDw7P9
2AjD2Apim4aKM7cNj8AiTmBs396P4KxHtzlw4SjKc9sIyZAl8AQMNOBKyHQ+HuNABPMZASsaUYh6
uBA086ez1YWBjCdTSeQYzuPVw9k/+uO1B/76Hh61Cl6kwgCffvX6VxL44QcZwcHHLl6NtFCbfTJa
vAbT30fn4nI+kwvk0+E489vub2DIVhdgaH5EHs6SSXamQ25gTGfv1w6RET6uDqi9FdSibqin2TLk
oUFnmgS/vW8SwU1NgutDfQSCb4Q1mLlG/Df/uexdyiTBADwItpkcR482IY3lRyLVYTJKbggIbAxf
nJF0IeqM6Fh8XXI/uZ2PwEd2TBZRLwgNm0rH4BYDCzvwTUOYJpiwPKBWVMfE5nk1qly+NWA1ggtQ
qJ+j4NOJ401qi+NN5jMMDU3Ww7VpVG89nDe+fwFKAkRtuB7bU2HcRMoPL+z1+F51+tl50TBd+h1i
q6cLTo+J4T6zZ+ZE+2rAuHTSbmyFYJpQ9Ml8MNklF0bgcgQFbHnwyKzYD/+++T9bDvmjM0qnHprc
32M+T5esTHuwt+vEq5QFrodbMQu8HqSOaZMfCuIarfQ6OaWo3a7HJLl2e01gq2UrzA5Jg7kr2NeO
n4Akv05NVQT7ObmajwI5BcG7smCVvD97KPC4q1ylKFvg8auth4oUc8ssHH6d9VCBXBy9kgbUQ7WE
bQu11gb/lSqHqpA5VuHuPeVQXccx7XTROM+UKLJwiGjxaoujX+SiMdLFaWKFSeoIz16uGtcx+LlI
NrTCpLC63G0rTObRRbBGVph0HIcvq8x7VZkVkXziCpNF4TXzag1/eU58dfa1aZ4YOqC7vp95hbAs
phx2Vr2VixBdVTCllVybdMmNdD255OLCFctCzlxz8P99ff36ugiStsl0JFcmjpYCKkd0jVaiS9Em
mY9IDIcQGVyYCbNNGsvrug9HxvIjPqlXbNScri3aGvq5dPGaaDSCFmaKm4VTudvLViSbqocR3lwP
h9YD7910Ekq1uJmFszA7YjEbR/0Fs/qLkP/YX7i2Db9ofyEl/IA/L/wJT/UXvtdfUDjh40n+tr+I
4Au1Xn1rXMNBE64M8M5zOOu86y+CsD+D39GUrn1Y4K7/7Dz7P+hr7q8NgQUA
--=_446d2e46938523efd4a4a0e83ab083c6--
11:42 a.m.
On 14 Apr 2016, at 09:57, nicolas(a)devels.es wrote:
Ok, that makes sense, thanks for the insight both Alex and Fred. I'm attaching the
VDSM log of the SPM node at the time of the pause. I couldn't find anything that would
clearly identify the problem, but maybe you'll be able to.
In extreme conditions it will happen. When your storage is slow to respond to extension
request, and when your write rate is very high then it may happen, as it is happening to
you, that you run out space sooner than the extension finishes. You can change the
watermark value I guess(right, Fred?), but better would be to plan a bit more ahead and
either use preallocated or create thin and then allocate expected size in advance before
the operation causing it (typically it only happens during untarring gigabytes of data, or
huge database dump/restore)
Even then, the VM should always be automatially resumed once the disk space is allocated
Thanks,
michal
Thanks.
Regards.
El 2016-04-13 13:09, Fred Rolland escribió:
> Hi,
> Yes, just as Alex explained, if the disk has been created as thin
> provisioning, the vdsm will extends once a watermark is reached.
> Usually it should not get to the state the Vm is paused.
> From the log, you can see that the request for extension has been sent
> before the VM got to the No Space Error.
> Later, we can see the VM resuming.
> INFO::2016-04-13
> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for
> volume
> ....
> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
> virtio-disk0 error enospc
> ....
> INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
> Note that the extension is done on the SPM host, so it would be
> interesting to see the vdsm log from the host that was in SPM role at
> this timeframe.
> Regards,
> Fred
> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow <acrow(a)integrafin.co.uk>
> wrote:
>> Hi,
>> If you have set up VM disks as Thin Provisioned, the VM has to
>> pause when the disk image needs to expand. You won't see this on VMs
>> with preallocated storage.
>> It's not the SAN that's running out of space, it's the VM image
>> needing to be expanded incrementally each time.
>> Cheers
>> Alex
>> On 13/04/16 12:04, nicolas(a)devels.es wrote:
>> Hi Fred,
>> This is an iSCSI storage. I'm attaching the VDSM logs from the host
>> where this machine has been running. Should you need any further
>> info, don't hesitate to ask.
>> Thanks.
>> Regards.
>> El 2016-04-13 11:54, Fred Rolland escribió:
>> Hi,
>> What kind of storage do you have ? (ISCSI,FC,NFS...)
>> Can you provide the vdsm logs from the host where this VM runs ?
>> Thanks,
>> Freddy
>> On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es> wrote:
>> Hi,
>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
>> events like these:
>> 2016-04-13 10:52:30,735 INFO
>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
>> from
>> 'Up' --> 'Paused'
>> 2016-04-13 10:52:30,815 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>> [1] [1]
>> has been paused.
>> 2016-04-13 10:52:30,898 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>> [1] [1]
>> has been paused due to no Storage space error.
>> 2016-04-13 10:52:52,320 WARN
>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
>> (org.ovirt.thread.pool-8-thread-38) [] domain
>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
>> 'host6.domain.com [2] [2]'
>> 2016-04-13 10:52:55,183 INFO
>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
>> from
>> 'Paused' --> 'Up'
>> 2016-04-13 10:52:55,318 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>> [1] [1]
>> has recovered from paused back to up.
>> The storage domain is far from being full, though (400+ G available
>> right now). Could this be related to this other issue [1]? If not,
>> how could I debug what's going on?
>> Thanks.
>> [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
>> [3]
>> [3]
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users [4] [4]
>> Links:
>> ------
>> [1] http://vm.domain.com [1]
>> [2] http://host6.domain.com [2]
>> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html [3]
>> [4] http://lists.ovirt.org/mailman/listinfo/users [4]
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users [4]
> --
> This message is intended only for the addressee and may contain
> confidential information. Unless you are that person, you may not
> disclose its contents or use it in any way and are requested to delete
> the message along with any attachments and notify us immediately.
> This email is not intended to, nor should it be taken to, constitute
> advice.
> The information provided is correct to our knowledge & belief and must
> not
> be used as a substitute for obtaining tax, regulatory, investment,
> legal or
> any other appropriate advice.
> "Transact" is operated by Integrated Financial Arrangements Ltd.
> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020)
> 7608 5300.
> (Registered office: as above; Registered in England and Wales under
> number: 3727592). Authorised and regulated by the Financial Conduct
> Authority (entered on the Financial Services Register; no. 190856).
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users [4]
> Links:
> ------
> [1] http://vm.domain.com
> [2] http://host6.domain.com
> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
> [4] http://lists.ovirt.org/mailman/listinfo/users
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
<vdsm.log.gz>_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
12:02 p.m.
From the log, we can see that the lvextend command took 18 sec, which
is
quite long.
60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' devices
{ preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/36000eb3a4f1acbc20000000000000043|'\'',
'\''r|.*|'\'' ]
} global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } '
--autobackup n --size 6016m
5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
(cwd None)
....
60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '
WARNING: lvmetad is running but disabled. Restart lvmetad before enabling
it!\n WARNING: This metadata update is NOT backed up\n'; <rc> = 0
The watermark can be configured by the following value:
'volume_utilization_percent', '50',
'Together with volume_utilization_chunk_mb, set the minimal free '
'space before a thin provisioned block volume is extended. Use '
'lower values to extend earlier.')
On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek <
michal.skrivanek(a)redhat.com> wrote:
> On 14 Apr 2016, at 09:57, nicolas(a)devels.es wrote:
>
> Ok, that makes sense, thanks for the insight both Alex and Fred. I'm
attaching the VDSM log of the SPM node at the time of the pause. I couldn't
find anything that would clearly identify the problem, but maybe you'll be
able to.
In extreme conditions it will happen. When your storage is slow to respond
to extension request, and when your write rate is very high then it may
happen, as it is happening to you, that you run out space sooner than the
extension finishes. You can change the watermark value I guess(right,
Fred?), but better would be to plan a bit more ahead and either use
preallocated or create thin and then allocate expected size in advance
before the operation causing it (typically it only happens during untarring
gigabytes of data, or huge database dump/restore)
Even then, the VM should always be automatially resumed once the disk
space is allocated
Thanks,
michal
>
> Thanks.
>
> Regards.
>
> El 2016-04-13 13:09, Fred Rolland escribió:
>> Hi,
>> Yes, just as Alex explained, if the disk has been created as thin
>> provisioning, the vdsm will extends once a watermark is reached.
>> Usually it should not get to the state the Vm is paused.
>> From the log, you can see that the request for extension has been sent
>> before the VM got to the No Space Error.
>> Later, we can see the VM resuming.
>> INFO::2016-04-13
>> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
>> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for
>> volume
>> ....
>> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
>> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
>> virtio-disk0 error enospc
>> ....
>> INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
>> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
>> Note that the extension is done on the SPM host, so it would be
>> interesting to see the vdsm log from the host that was in SPM role at
>> this timeframe.
>> Regards,
>> Fred
>> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow <acrow(a)integrafin.co.uk>
>> wrote:
>>> Hi,
>>> If you have set up VM disks as Thin Provisioned, the VM has to
>>> pause when the disk image needs to expand. You won't see this on VMs
>>> with preallocated storage.
>>> It's not the SAN that's running out of space, it's the VM image
>>> needing to be expanded incrementally each time.
>>> Cheers
>>> Alex
>>> On 13/04/16 12:04, nicolas(a)devels.es wrote:
>>> Hi Fred,
>>> This is an iSCSI storage. I'm attaching the VDSM logs from the host
>>> where this machine has been running. Should you need any further
>>> info, don't hesitate to ask.
>>> Thanks.
>>> Regards.
>>> El 2016-04-13 11:54, Fred Rolland escribió:
>>> Hi,
>>> What kind of storage do you have ? (ISCSI,FC,NFS...)
>>> Can you provide the vdsm logs from the host where this VM runs ?
>>> Thanks,
>>> Freddy
>>> On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es> wrote:
>>> Hi,
>>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
>>> events like these:
>>> 2016-04-13 10:52:30,735 INFO
>>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
>>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
>>> from
>>> 'Up' --> 'Paused'
>>> 2016-04-13 10:52:30,815 INFO
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
>>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>>> [1] [1]
>>> has been paused.
>>> 2016-04-13 10:52:30,898 ERROR
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
>>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>>> [1] [1]
>>> has been paused due to no Storage space error.
>>> 2016-04-13 10:52:52,320 WARN
>>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
>>> (org.ovirt.thread.pool-8-thread-38) [] domain
>>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
>>> 'host6.domain.com [2] [2]'
>>> 2016-04-13 10:52:55,183 INFO
>>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
>>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
>>> from
>>> 'Paused' --> 'Up'
>>> 2016-04-13 10:52:55,318 INFO
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
>>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>>> [1] [1]
>>> has recovered from paused back to up.
>>> The storage domain is far from being full, though (400+ G available
>>> right now). Could this be related to this other issue [1]? If not,
>>> how could I debug what's going on?
>>> Thanks.
>>> [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
>>> [3]
>>> [3]
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users [4] [4]
>>> Links:
>>> ------
>>> [1] http://vm.domain.com [1]
>>> [2] http://host6.domain.com [2]
>>> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html [3]
>>> [4] http://lists.ovirt.org/mailman/listinfo/users [4]
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users [4]
>> --
>> This message is intended only for the addressee and may contain
>> confidential information. Unless you are that person, you may not
>> disclose its contents or use it in any way and are requested to delete
>> the message along with any attachments and notify us immediately.
>> This email is not intended to, nor should it be taken to, constitute
>> advice.
>> The information provided is correct to our knowledge & belief and must
>> not
>> be used as a substitute for obtaining tax, regulatory, investment,
>> legal or
>> any other appropriate advice.
>> "Transact" is operated by Integrated Financial Arrangements Ltd.
>> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020)
>> 7608 5300.
>> (Registered office: as above; Registered in England and Wales under
>> number: 3727592). Authorised and regulated by the Financial Conduct
>> Authority (entered on the Financial Services Register; no. 190856).
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users [4]
>> Links:
>> ------
>> [1] http://vm.domain.com
>> [2] http://host6.domain.com
>> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
>> [4] http://lists.ovirt.org/mailman/listinfo/users
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
> <vdsm.log.gz>_______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
12:18 p.m.
On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland <frolland(a)redhat.com> wrote:
From the log, we can see that the lvextend command took 18 sec, which
is
quite long.
Fred, can you run repoplot on this log file? it will may explain why this lvm
call took 18 seconds.
Nir
60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' devices {
preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/36000eb3a4f1acbc20000000000000043|'\'',
'\''r|.*|'\'' ] }
global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --autobackup
n --size 6016m
5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
(cwd None)
....
60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '
WARNING: lvmetad is running but disabled. Restart lvmetad before enabling
it!\n WARNING: This metadata update is NOT backed up\n'; <rc> = 0
The watermark can be configured by the following value:
'volume_utilization_percent', '50',
'Together with volume_utilization_chunk_mb, set the minimal free '
'space before a thin provisioned block volume is extended. Use '
'lower values to extend earlier.')
On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek
<michal.skrivanek(a)redhat.com> wrote:
>
>
> > On 14 Apr 2016, at 09:57, nicolas(a)devels.es wrote:
> >
> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm
> > attaching the VDSM log of the SPM node at the time of the pause. I couldn't
> > find anything that would clearly identify the problem, but maybe you'll be
> > able to.
>
> In extreme conditions it will happen. When your storage is slow to respond
> to extension request, and when your write rate is very high then it may
> happen, as it is happening to you, that you run out space sooner than the
> extension finishes. You can change the watermark value I guess(right,
> Fred?), but better would be to plan a bit more ahead and either use
> preallocated or create thin and then allocate expected size in advance
> before the operation causing it (typically it only happens during untarring
> gigabytes of data, or huge database dump/restore)
> Even then, the VM should always be automatially resumed once the disk
> space is allocated
>
> Thanks,
> michal
>
> >
> > Thanks.
> >
> > Regards.
> >
> > El 2016-04-13 13:09, Fred Rolland escribió:
> >> Hi,
> >> Yes, just as Alex explained, if the disk has been created as thin
> >> provisioning, the vdsm will extends once a watermark is reached.
> >> Usually it should not get to the state the Vm is paused.
> >> From the log, you can see that the request for extension has been sent
> >> before the VM got to the No Space Error.
> >> Later, we can see the VM resuming.
> >> INFO::2016-04-13
> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for
> >> volume
> >> ....
> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
> >> virtio-disk0 error enospc
> >> ....
> >> INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
> >> Note that the extension is done on the SPM host, so it would be
> >> interesting to see the vdsm log from the host that was in SPM role at
> >> this timeframe.
> >> Regards,
> >> Fred
> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow <acrow(a)integrafin.co.uk>
> >> wrote:
> >>> Hi,
> >>> If you have set up VM disks as Thin Provisioned, the VM has to
> >>> pause when the disk image needs to expand. You won't see this on
VMs
> >>> with preallocated storage.
> >>> It's not the SAN that's running out of space, it's the VM
image
> >>> needing to be expanded incrementally each time.
> >>> Cheers
> >>> Alex
> >>> On 13/04/16 12:04, nicolas(a)devels.es wrote:
> >>> Hi Fred,
> >>> This is an iSCSI storage. I'm attaching the VDSM logs from the host
> >>> where this machine has been running. Should you need any further
> >>> info, don't hesitate to ask.
> >>> Thanks.
> >>> Regards.
> >>> El 2016-04-13 11:54, Fred Rolland escribió:
> >>> Hi,
> >>> What kind of storage do you have ? (ISCSI,FC,NFS...)
> >>> Can you provide the vdsm logs from the host where this VM runs ?
> >>> Thanks,
> >>> Freddy
> >>> On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es> wrote:
> >>> Hi,
> >>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
> >>> events like these:
> >>> 2016-04-13 10:52:30,735 INFO
> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
> >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1])
moved
> >>> from
> >>> 'Up' --> 'Paused'
> >>> 2016-04-13 10:52:30,815 INFO
> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
> >>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
> >>> [1] [1]
> >>> has been paused.
> >>> 2016-04-13 10:52:30,898 ERROR
> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
> >>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
> >>> [1] [1]
> >>> has been paused due to no Storage space error.
> >>> 2016-04-13 10:52:52,320 WARN
> >>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> >>> (org.ovirt.thread.pool-8-thread-38) [] domain
> >>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
> >>> 'host6.domain.com [2] [2]'
> >>> 2016-04-13 10:52:55,183 INFO
> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
> >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1])
moved
> >>> from
> >>> 'Paused' --> 'Up'
> >>> 2016-04-13 10:52:55,318 INFO
> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
> >>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
> >>> [1] [1]
> >>> has recovered from paused back to up.
> >>> The storage domain is far from being full, though (400+ G available
> >>> right now). Could this be related to this other issue [1]? If not,
> >>> how could I debug what's going on?
> >>> Thanks.
> >>> [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
> >>> [3]
> >>> [3]
> >>> _______________________________________________
> >>> Users mailing list
> >>> Users(a)ovirt.org
> >>> http://lists.ovirt.org/mailman/listinfo/users [4] [4]
> >>> Links:
> >>> ------
> >>> [1] http://vm.domain.com [1]
> >>> [2] http://host6.domain.com [2]
> >>> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html [3]
> >>> [4] http://lists.ovirt.org/mailman/listinfo/users [4]
> >> _______________________________________________
> >> Users mailing list
> >> Users(a)ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/users [4]
> >> --
> >> This message is intended only for the addressee and may contain
> >> confidential information. Unless you are that person, you may not
> >> disclose its contents or use it in any way and are requested to delete
> >> the message along with any attachments and notify us immediately.
> >> This email is not intended to, nor should it be taken to, constitute
> >> advice.
> >> The information provided is correct to our knowledge & belief and must
> >> not
> >> be used as a substitute for obtaining tax, regulatory, investment,
> >> legal or
> >> any other appropriate advice.
> >> "Transact" is operated by Integrated Financial Arrangements Ltd.
> >> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020)
> >> 7608 5300.
> >> (Registered office: as above; Registered in England and Wales under
> >> number: 3727592). Authorised and regulated by the Financial Conduct
> >> Authority (entered on the Financial Services Register; no. 190856).
> >> _______________________________________________
> >> Users mailing list
> >> Users(a)ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/users [4]
> >> Links:
> >> ------
> >> [1] http://vm.domain.com
> >> [2] http://host6.domain.com
> >> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
> >> [4] http://lists.ovirt.org/mailman/listinfo/users
> >> _______________________________________________
> >> Users mailing list
> >> Users(a)ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/users
> > <vdsm.log.gz>_______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
12:38 p.m.
Nir,
See attached the repoplot output.
On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland
<frolland(a)redhat.com>
wrote:
> From the log, we can see that the lvextend command took 18 sec, which is
> quite long.
Fred, can you run repoplot on this log file? it will may explain why this
lvm
call took 18 seconds.
Nir
>
> 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
> 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config '
devices {
> preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
> write_cache_state=0 disable_after_error_count=3 filter = [
> '\''a|/dev/mapper/36000eb3a4f1acbc20000000000000043|'\'',
'\''r|.*|'\''
] }
> global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } '
--autobackup
> n --size 6016m
> 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
> (cwd None)
> ....
> 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
> 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '
> WARNING: lvmetad is running but disabled. Restart lvmetad before enabling
> it!\n WARNING: This metadata update is NOT backed up\n'; <rc> = 0
>
>
> The watermark can be configured by the following value:
>
> 'volume_utilization_percent', '50',
> 'Together with volume_utilization_chunk_mb, set the minimal free '
> 'space before a thin provisioned block volume is extended. Use '
> 'lower values to extend earlier.')
>
> On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek
> <michal.skrivanek(a)redhat.com> wrote:
>>
>>
>> > On 14 Apr 2016, at 09:57, nicolas(a)devels.es wrote:
>> >
>> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm
>> > attaching the VDSM log of the SPM node at the time of the pause. I
couldn't
>> > find anything that would clearly identify the problem, but maybe
you'll be
>> > able to.
>>
>> In extreme conditions it will happen. When your storage is slow to
respond
>> to extension request, and when your write rate is very high then it may
>> happen, as it is happening to you, that you run out space sooner than
the
>> extension finishes. You can change the watermark value I guess(right,
>> Fred?), but better would be to plan a bit more ahead and either use
>> preallocated or create thin and then allocate expected size in advance
>> before the operation causing it (typically it only happens during
untarring
>> gigabytes of data, or huge database dump/restore)
>> Even then, the VM should always be automatially resumed once the disk
>> space is allocated
>>
>> Thanks,
>> michal
>>
>> >
>> > Thanks.
>> >
>> > Regards.
>> >
>> > El 2016-04-13 13:09, Fred Rolland escribió:
>> >> Hi,
>> >> Yes, just as Alex explained, if the disk has been created as thin
>> >> provisioning, the vdsm will extends once a watermark is reached.
>> >> Usually it should not get to the state the Vm is paused.
>> >> From the log, you can see that the request for extension has been
sent
>> >> before the VM got to the No Space Error.
>> >> Later, we can see the VM resuming.
>> >> INFO::2016-04-13
>> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for
>> >> volume
>> >> ....
>> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
>> >> virtio-disk0 error enospc
>> >> ....
>> >> INFO::2016-04-13
10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
>> >> Note that the extension is done on the SPM host, so it would be
>> >> interesting to see the vdsm log from the host that was in SPM role at
>> >> this timeframe.
>> >> Regards,
>> >> Fred
>> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow
<acrow(a)integrafin.co.uk>
>> >> wrote:
>> >>> Hi,
>> >>> If you have set up VM disks as Thin Provisioned, the VM has to
>> >>> pause when the disk image needs to expand. You won't see this
on VMs
>> >>> with preallocated storage.
>> >>> It's not the SAN that's running out of space, it's the
VM image
>> >>> needing to be expanded incrementally each time.
>> >>> Cheers
>> >>> Alex
>> >>> On 13/04/16 12:04, nicolas(a)devels.es wrote:
>> >>> Hi Fred,
>> >>> This is an iSCSI storage. I'm attaching the VDSM logs from the
host
>> >>> where this machine has been running. Should you need any further
>> >>> info, don't hesitate to ask.
>> >>> Thanks.
>> >>> Regards.
>> >>> El 2016-04-13 11:54, Fred Rolland escribió:
>> >>> Hi,
>> >>> What kind of storage do you have ? (ISCSI,FC,NFS...)
>> >>> Can you provide the vdsm logs from the host where this VM runs ?
>> >>> Thanks,
>> >>> Freddy
>> >>> On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es> wrote:
>> >>> Hi,
>> >>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch
of
>> >>> events like these:
>> >>> 2016-04-13 10:52:30,735 INFO
>> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
>> >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]
[1]) moved
>> >>> from
>> >>> 'Up' --> 'Paused'
>> >>> 2016-04-13 10:52:30,815 INFO
>> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID:
null,
>> >>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>> >>> [1] [1]
>> >>> has been paused.
>> >>> 2016-04-13 10:52:30,898 ERROR
>> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID:
null,
>> >>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>> >>> [1] [1]
>> >>> has been paused due to no Storage space error.
>> >>> 2016-04-13 10:52:52,320 WARN
>> >>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
>> >>> (org.ovirt.thread.pool-8-thread-38) [] domain
>> >>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem.
vds:
>> >>> 'host6.domain.com [2] [2]'
>> >>> 2016-04-13 10:52:55,183 INFO
>> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
>> >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]
[1]) moved
>> >>> from
>> >>> 'Paused' --> 'Up'
>> >>> 2016-04-13 10:52:55,318 INFO
>> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID:
null,
>> >>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>> >>> [1] [1]
>> >>> has recovered from paused back to up.
>> >>> The storage domain is far from being full, though (400+ G
available
>> >>> right now). Could this be related to this other issue [1]? If not,
>> >>> how could I debug what's going on?
>> >>> Thanks.
>> >>> [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
>> >>> [3]
>> >>> [3]
>> >>> _______________________________________________
>> >>> Users mailing list
>> >>> Users(a)ovirt.org
>> >>> http://lists.ovirt.org/mailman/listinfo/users [4] [4]
>> >>> Links:
>> >>> ------
>> >>> [1] http://vm.domain.com [1]
>> >>> [2] http://host6.domain.com [2]
>> >>> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html [3]
>> >>> [4] http://lists.ovirt.org/mailman/listinfo/users [4]
>> >> _______________________________________________
>> >> Users mailing list
>> >> Users(a)ovirt.org
>> >> http://lists.ovirt.org/mailman/listinfo/users [4]
>> >> --
>> >> This message is intended only for the addressee and may contain
>> >> confidential information. Unless you are that person, you may not
>> >> disclose its contents or use it in any way and are requested to
delete
>> >> the message along with any attachments and notify us immediately.
>> >> This email is not intended to, nor should it be taken to, constitute
>> >> advice.
>> >> The information provided is correct to our knowledge & belief and
must
>> >> not
>> >> be used as a substitute for obtaining tax, regulatory, investment,
>> >> legal or
>> >> any other appropriate advice.
>> >> "Transact" is operated by Integrated Financial Arrangements
Ltd.
>> >> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax:
(020)
>> >> 7608 5300.
>> >> (Registered office: as above; Registered in England and Wales under
>> >> number: 3727592). Authorised and regulated by the Financial Conduct
>> >> Authority (entered on the Financial Services Register; no. 190856).
>> >> _______________________________________________
>> >> Users mailing list
>> >> Users(a)ovirt.org
>> >> http://lists.ovirt.org/mailman/listinfo/users [4]
>> >> Links:
>> >> ------
>> >> [1] http://vm.domain.com
>> >> [2] http://host6.domain.com
>> >> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
>> >> [4] http://lists.ovirt.org/mailman/listinfo/users
>> >> _______________________________________________
>> >> Users mailing list
>> >> Users(a)ovirt.org
>> >> http://lists.ovirt.org/mailman/listinfo/users
>> > <vdsm.log.gz>_______________________________________________
>> > Users mailing list
>> > Users(a)ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
1:02 p.m.
On Thu, Apr 14, 2016 at 12:38 PM, Fred Rolland <frolland(a)redhat.com> wrote:
Nir,
See attached the repoplot output.
So we have about one concurrent lvm command without any disk operations, and
everything seems snappy.
Nicolás, maybe this storage or the host is overloaded by the vms? Are your vms
doing lot of io?
lvextend operation should be very fast operation, this is just a
metadata change,
allocating couple of extents to that lv.
Zdenek, how do you suggest to debug slow lvm commands?
See the attached pdf, lvm commands took 15-50 seconds.
On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
>
> On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland <frolland(a)redhat.com>
> wrote:
> > From the log, we can see that the lvextend command took 18 sec, which is
> > quite long.
>
> Fred, can you run repoplot on this log file? it will may explain why this
> lvm
> call took 18 seconds.
>
> Nir
>
> >
> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
> > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config '
> > devices {
> > preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3 filter = [
> >
'\''a|/dev/mapper/36000eb3a4f1acbc20000000000000043|'\'',
'\''r|.*|'\''
> > ] }
> > global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> > use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } '
> > --autobackup
> > n --size 6016m
> >
> > 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
> > (cwd None)
> > ....
> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
> > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '
> > WARNING: lvmetad is running but disabled. Restart lvmetad before
> > enabling
> > it!\n WARNING: This metadata update is NOT backed up\n'; <rc> = 0
> >
> >
> > The watermark can be configured by the following value:
> >
> > 'volume_utilization_percent', '50',
> > 'Together with volume_utilization_chunk_mb, set the minimal free '
> > 'space before a thin provisioned block volume is extended. Use '
> > 'lower values to extend earlier.')
> >
> > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek
> > <michal.skrivanek(a)redhat.com> wrote:
> >>
> >>
> >> > On 14 Apr 2016, at 09:57, nicolas(a)devels.es wrote:
> >> >
> >> > Ok, that makes sense, thanks for the insight both Alex and Fred.
I'm
> >> > attaching the VDSM log of the SPM node at the time of the pause. I
> >> > couldn't
> >> > find anything that would clearly identify the problem, but maybe
> >> > you'll be
> >> > able to.
> >>
> >> In extreme conditions it will happen. When your storage is slow to
> >> respond
> >> to extension request, and when your write rate is very high then it may
> >> happen, as it is happening to you, that you run out space sooner than
> >> the
> >> extension finishes. You can change the watermark value I guess(right,
> >> Fred?), but better would be to plan a bit more ahead and either use
> >> preallocated or create thin and then allocate expected size in advance
> >> before the operation causing it (typically it only happens during
> >> untarring
> >> gigabytes of data, or huge database dump/restore)
> >> Even then, the VM should always be automatially resumed once the disk
> >> space is allocated
> >>
> >> Thanks,
> >> michal
> >>
> >> >
> >> > Thanks.
> >> >
> >> > Regards.
> >> >
> >> > El 2016-04-13 13:09, Fred Rolland escribió:
> >> >> Hi,
> >> >> Yes, just as Alex explained, if the disk has been created as thin
> >> >> provisioning, the vdsm will extends once a watermark is reached.
> >> >> Usually it should not get to the state the Vm is paused.
> >> >> From the log, you can see that the request for extension has been
> >> >> sent
> >> >> before the VM got to the No Space Error.
> >> >> Later, we can see the VM resuming.
> >> >> INFO::2016-04-13
> >> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension
> >> >> for
> >> >> volume
> >> >> ....
> >> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop
device
> >> >> virtio-disk0 error enospc
> >> >> ....
> >> >> INFO::2016-04-13
> >> >> 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
> >> >> Note that the extension is done on the SPM host, so it would be
> >> >> interesting to see the vdsm log from the host that was in SPM role
> >> >> at
> >> >> this timeframe.
> >> >> Regards,
> >> >> Fred
> >> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow
<acrow(a)integrafin.co.uk>
> >> >> wrote:
> >> >>> Hi,
> >> >>> If you have set up VM disks as Thin Provisioned, the VM has to
> >> >>> pause when the disk image needs to expand. You won't see
this on
> >> >>> VMs
> >> >>> with preallocated storage.
> >> >>> It's not the SAN that's running out of space, it's
the VM image
> >> >>> needing to be expanded incrementally each time.
> >> >>> Cheers
> >> >>> Alex
> >> >>> On 13/04/16 12:04, nicolas(a)devels.es wrote:
> >> >>> Hi Fred,
> >> >>> This is an iSCSI storage. I'm attaching the VDSM logs from
the host
> >> >>> where this machine has been running. Should you need any
further
> >> >>> info, don't hesitate to ask.
> >> >>> Thanks.
> >> >>> Regards.
> >> >>> El 2016-04-13 11:54, Fred Rolland escribió:
> >> >>> Hi,
> >> >>> What kind of storage do you have ? (ISCSI,FC,NFS...)
> >> >>> Can you provide the vdsm logs from the host where this VM runs
?
> >> >>> Thanks,
> >> >>> Freddy
> >> >>> On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es>
wrote:
> >> >>> Hi,
> >> >>> We're running oVirt 3.6.4.1-1. Lately we're seeing a
bunch of
> >> >>> events like these:
> >> >>> 2016-04-13 10:52:30,735 INFO
> >> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> >> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
> >> >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com
[1] [1]) moved
> >> >>> from
> >> >>> 'Up' --> 'Paused'
> >> >>> 2016-04-13 10:52:30,815 INFO
> >> >>
> >> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID:
null,
> >> >>> Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com
> >> >>> [1] [1]
> >> >>> has been paused.
> >> >>> 2016-04-13 10:52:30,898 ERROR
> >> >>
> >> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID:
null,
> >> >>> Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com
> >> >>> [1] [1]
> >> >>> has been paused due to no Storage space error.
> >> >>> 2016-04-13 10:52:52,320 WARN
> >> >>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> >> >>> (org.ovirt.thread.pool-8-thread-38) [] domain
> >> >>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in
problem. vds:
> >> >>> 'host6.domain.com [2] [2]'
> >> >>> 2016-04-13 10:52:55,183 INFO
> >> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> >> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
> >> >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com
[1] [1]) moved
> >> >>> from
> >> >>> 'Paused' --> 'Up'
> >> >>> 2016-04-13 10:52:55,318 INFO
> >> >>
> >> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID:
null,
> >> >>> Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com
> >> >>> [1] [1]
> >> >>> has recovered from paused back to up.
> >> >>> The storage domain is far from being full, though (400+ G
available
> >> >>> right now). Could this be related to this other issue [1]? If
not,
> >> >>> how could I debug what's going on?
> >> >>> Thanks.
> >> >>> [1]:
https://www.mail-archive.com/users@ovirt.org/msg32079.html
> >> >>> [3]
> >> >>> [3]
> >> >>> _______________________________________________
> >> >>> Users mailing list
> >> >>> Users(a)ovirt.org
> >> >>> http://lists.ovirt.org/mailman/listinfo/users [4] [4]
> >> >>> Links:
> >> >>> ------
> >> >>> [1] http://vm.domain.com [1]
> >> >>> [2] http://host6.domain.com [2]
> >> >>> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
[3]
> >> >>> [4] http://lists.ovirt.org/mailman/listinfo/users [4]
> >> >> _______________________________________________
> >> >> Users mailing list
> >> >> Users(a)ovirt.org
> >> >> http://lists.ovirt.org/mailman/listinfo/users [4]
> >> >> --
> >> >> This message is intended only for the addressee and may contain
> >> >> confidential information. Unless you are that person, you may not
> >> >> disclose its contents or use it in any way and are requested to
> >> >> delete
> >> >> the message along with any attachments and notify us immediately.
> >> >> This email is not intended to, nor should it be taken to,
constitute
> >> >> advice.
> >> >> The information provided is correct to our knowledge & belief
and
> >> >> must
> >> >> not
> >> >> be used as a substitute for obtaining tax, regulatory, investment,
> >> >> legal or
> >> >> any other appropriate advice.
> >> >> "Transact" is operated by Integrated Financial
Arrangements Ltd.
> >> >> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax:
(020)
> >> >> 7608 5300.
> >> >> (Registered office: as above; Registered in England and Wales
under
> >> >> number: 3727592). Authorised and regulated by the Financial
Conduct
> >> >> Authority (entered on the Financial Services Register; no.
190856).
> >> >> _______________________________________________
> >> >> Users mailing list
> >> >> Users(a)ovirt.org
> >> >> http://lists.ovirt.org/mailman/listinfo/users [4]
> >> >> Links:
> >> >> ------
> >> >> [1] http://vm.domain.com
> >> >> [2] http://host6.domain.com
> >> >> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
> >> >> [4] http://lists.ovirt.org/mailman/listinfo/users
> >> >> _______________________________________________
> >> >> Users mailing list
> >> >> Users(a)ovirt.org
> >> >> http://lists.ovirt.org/mailman/listinfo/users
> >> > <vdsm.log.gz>_______________________________________________
> >> > Users mailing list
> >> > Users(a)ovirt.org
> >> > http://lists.ovirt.org/mailman/listinfo/users
> >>
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
1:23 p.m.
Hi Nir,
El 2016-04-14 11:02, Nir Soffer escribió:
On Thu, Apr 14, 2016 at 12:38 PM, Fred Rolland
<frolland(a)redhat.com>
wrote:
> Nir,
> See attached the repoplot output.
So we have about one concurrent lvm command without any disk
operations, and
everything seems snappy.
Nicolás, maybe this storage or the host is overloaded by the vms? Are
your vms
doing lot of io?
Not that I know, actually it should have been a "calm" time slot as far
as IOs go, nor the storage was overloaded at that time. If I'm not
mistaken, on the repoplot report I see there are two LVM operations at a
time, maybe that has something to do with it? (although as you say, the
lvextend is just a metadata change...)
lvextend operation should be very fast operation, this is just a
metadata change,
allocating couple of extents to that lv.
Zdenek, how do you suggest to debug slow lvm commands?
See the attached pdf, lvm commands took 15-50 seconds.
>
> On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer <nsoffer(a)redhat.com>
> wrote:
>>
>> On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland <frolland(a)redhat.com>
>> wrote:
>> > From the log, we can see that the lvextend command took 18 sec, which is
>> > quite long.
>>
>> Fred, can you run repoplot on this log file? it will may explain why
>> this
>> lvm
>> call took 18 seconds.
>>
>> Nir
>>
>> >
>> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
>> > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
>> > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config '
>> > devices {
>> > preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
>> > write_cache_state=0 disable_after_error_count=3 filter = [
>> >
'\''a|/dev/mapper/36000eb3a4f1acbc20000000000000043|'\'',
'\''r|.*|'\''
>> > ] }
>> > global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1
>> > use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } '
>> > --autobackup
>> > n --size 6016m
>> >
>> > 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
>> > (cwd None)
>> > ....
>> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
>> > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> =
'
>> > WARNING: lvmetad is running but disabled. Restart lvmetad before
>> > enabling
>> > it!\n WARNING: This metadata update is NOT backed up\n'; <rc> =
0
>> >
>> >
>> > The watermark can be configured by the following value:
>> >
>> > 'volume_utilization_percent', '50',
>> > 'Together with volume_utilization_chunk_mb, set the minimal free
'
>> > 'space before a thin provisioned block volume is extended. Use
'
>> > 'lower values to extend earlier.')
>> >
>> > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek
>> > <michal.skrivanek(a)redhat.com> wrote:
>> >>
>> >>
>> >> > On 14 Apr 2016, at 09:57, nicolas(a)devels.es wrote:
>> >> >
>> >> > Ok, that makes sense, thanks for the insight both Alex and Fred.
I'm
>> >> > attaching the VDSM log of the SPM node at the time of the pause. I
>> >> > couldn't
>> >> > find anything that would clearly identify the problem, but maybe
>> >> > you'll be
>> >> > able to.
>> >>
>> >> In extreme conditions it will happen. When your storage is slow to
>> >> respond
>> >> to extension request, and when your write rate is very high then it may
>> >> happen, as it is happening to you, that you run out space sooner than
>> >> the
>> >> extension finishes. You can change the watermark value I guess(right,
>> >> Fred?), but better would be to plan a bit more ahead and either use
>> >> preallocated or create thin and then allocate expected size in advance
>> >> before the operation causing it (typically it only happens during
>> >> untarring
>> >> gigabytes of data, or huge database dump/restore)
>> >> Even then, the VM should always be automatially resumed once the disk
>> >> space is allocated
>> >>
>> >> Thanks,
>> >> michal
>> >>
>> >> >
>> >> > Thanks.
>> >> >
>> >> > Regards.
>> >> >
>> >> > El 2016-04-13 13:09, Fred Rolland escribió:
>> >> >> Hi,
>> >> >> Yes, just as Alex explained, if the disk has been created as
thin
>> >> >> provisioning, the vdsm will extends once a watermark is
reached.
>> >> >> Usually it should not get to the state the Vm is paused.
>> >> >> From the log, you can see that the request for extension has
been
>> >> >> sent
>> >> >> before the VM got to the No Space Error.
>> >> >> Later, we can see the VM resuming.
>> >> >> INFO::2016-04-13
>> >> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
>> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting
extension
>> >> >> for
>> >> >> volume
>> >> >> ....
>> >> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
>> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop
device
>> >> >> virtio-disk0 error enospc
>> >> >> ....
>> >> >> INFO::2016-04-13
>> >> >> 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
>> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running:
onResume
>> >> >> Note that the extension is done on the SPM host, so it would
be
>> >> >> interesting to see the vdsm log from the host that was in SPM
role
>> >> >> at
>> >> >> this timeframe.
>> >> >> Regards,
>> >> >> Fred
>> >> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow
<acrow(a)integrafin.co.uk>
>> >> >> wrote:
>> >> >>> Hi,
>> >> >>> If you have set up VM disks as Thin Provisioned, the VM has
to
>> >> >>> pause when the disk image needs to expand. You won't
see this on
>> >> >>> VMs
>> >> >>> with preallocated storage.
>> >> >>> It's not the SAN that's running out of space,
it's the VM image
>> >> >>> needing to be expanded incrementally each time.
>> >> >>> Cheers
>> >> >>> Alex
>> >> >>> On 13/04/16 12:04, nicolas(a)devels.es wrote:
>> >> >>> Hi Fred,
>> >> >>> This is an iSCSI storage. I'm attaching the VDSM logs
from the host
>> >> >>> where this machine has been running. Should you need any
further
>> >> >>> info, don't hesitate to ask.
>> >> >>> Thanks.
>> >> >>> Regards.
>> >> >>> El 2016-04-13 11:54, Fred Rolland escribió:
>> >> >>> Hi,
>> >> >>> What kind of storage do you have ? (ISCSI,FC,NFS...)
>> >> >>> Can you provide the vdsm logs from the host where this VM
runs ?
>> >> >>> Thanks,
>> >> >>> Freddy
>> >> >>> On Wed, Apr 13, 2016 at 1:02 PM, <nicolas(a)devels.es>
wrote:
>> >> >>> Hi,
>> >> >>> We're running oVirt 3.6.4.1-1. Lately we're seeing
a bunch of
>> >> >>> events like these:
>> >> >>> 2016-04-13 10:52:30,735 INFO
>> >> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>> >> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
>> >> >>>
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
>> >> >>> from
>> >> >>> 'Up' --> 'Paused'
>> >> >>> 2016-04-13 10:52:30,815 INFO
>> >> >>
>> >> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> >> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation
ID: null,
>> >> >>> Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com
>> >> >>> [1] [1]
>> >> >>> has been paused.
>> >> >>> 2016-04-13 10:52:30,898 ERROR
>> >> >>
>> >> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> >> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation
ID: null,
>> >> >>> Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com
>> >> >>> [1] [1]
>> >> >>> has been paused due to no Storage space error.
>> >> >>> 2016-04-13 10:52:52,320 WARN
>> >> >>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
>> >> >>> (org.ovirt.thread.pool-8-thread-38) [] domain
>> >> >>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in
problem. vds:
>> >> >>> 'host6.domain.com [2] [2]'
>> >> >>> 2016-04-13 10:52:55,183 INFO
>> >> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>> >> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
>> >> >>>
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
>> >> >>> from
>> >> >>> 'Paused' --> 'Up'
>> >> >>> 2016-04-13 10:52:55,318 INFO
>> >> >>
>> >> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> >> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation
ID: null,
>> >> >>> Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com
>> >> >>> [1] [1]
>> >> >>> has recovered from paused back to up.
>> >> >>> The storage domain is far from being full, though (400+ G
available
>> >> >>> right now). Could this be related to this other issue [1]?
If not,
>> >> >>> how could I debug what's going on?
>> >> >>> Thanks.
>> >> >>> [1]:
https://www.mail-archive.com/users@ovirt.org/msg32079.html
>> >> >>> [3]
>> >> >>> [3]
>> >> >>> _______________________________________________
>> >> >>> Users mailing list
>> >> >>> Users(a)ovirt.org
>> >> >>> http://lists.ovirt.org/mailman/listinfo/users [4] [4]
>> >> >>> Links:
>> >> >>> ------
>> >> >>> [1] http://vm.domain.com [1]
>> >> >>> [2] http://host6.domain.com [2]
>> >> >>> [3]
https://www.mail-archive.com/users@ovirt.org/msg32079.html [3]
>> >> >>> [4] http://lists.ovirt.org/mailman/listinfo/users [4]
>> >> >> _______________________________________________
>> >> >> Users mailing list
>> >> >> Users(a)ovirt.org
>> >> >> http://lists.ovirt.org/mailman/listinfo/users [4]
>> >> >> --
>> >> >> This message is intended only for the addressee and may
contain
>> >> >> confidential information. Unless you are that person, you may
not
>> >> >> disclose its contents or use it in any way and are requested
to
>> >> >> delete
>> >> >> the message along with any attachments and notify us
immediately.
>> >> >> This email is not intended to, nor should it be taken to,
constitute
>> >> >> advice.
>> >> >> The information provided is correct to our knowledge &
belief and
>> >> >> must
>> >> >> not
>> >> >> be used as a substitute for obtaining tax, regulatory,
investment,
>> >> >> legal or
>> >> >> any other appropriate advice.
>> >> >> "Transact" is operated by Integrated Financial
Arrangements Ltd.
>> >> >> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900
Fax: (020)
>> >> >> 7608 5300.
>> >> >> (Registered office: as above; Registered in England and Wales
under
>> >> >> number: 3727592). Authorised and regulated by the Financial
Conduct
>> >> >> Authority (entered on the Financial Services Register; no.
190856).
>> >> >> _______________________________________________
>> >> >> Users mailing list
>> >> >> Users(a)ovirt.org
>> >> >> http://lists.ovirt.org/mailman/listinfo/users [4]
>> >> >> Links:
>> >> >> ------
>> >> >> [1] http://vm.domain.com
>> >> >> [2] http://host6.domain.com
>> >> >> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
>> >> >> [4] http://lists.ovirt.org/mailman/listinfo/users
>> >> >> _______________________________________________
>> >> >> Users mailing list
>> >> >> Users(a)ovirt.org
>> >> >> http://lists.ovirt.org/mailman/listinfo/users
>> >> > <vdsm.log.gz>_______________________________________________
>> >> > Users mailing list
>> >> > Users(a)ovirt.org
>> >> > http://lists.ovirt.org/mailman/listinfo/users
>> >>
>> >
>> >
>> > _______________________________________________
>> > Users mailing list
>> > Users(a)ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>> >
>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
1:29 p.m.
On Thu, Apr 14, 2016 at 1:23 PM, <nicolas(a)devels.es> wrote:
Hi Nir,
El 2016-04-14 11:02, Nir Soffer escribió:
>
> On Thu, Apr 14, 2016 at 12:38 PM, Fred Rolland <frolland(a)redhat.com>
> wrote:
>>
>> Nir,
>> See attached the repoplot output.
>
>
> So we have about one concurrent lvm command without any disk operations,
> and
> everything seems snappy.
>
> Nicolás, maybe this storage or the host is overloaded by the vms? Are your
> vms
> doing lot of io?
>
Not that I know, actually it should have been a "calm" time slot as far as
IOs go, nor the storage was overloaded at that time. If I'm not mistaken, on
the repoplot report I see there are two LVM operations at a time, maybe that
has something to do with it?
The operation that took about 50 seconds started in the same time that another
operation started, but it does not explain why several other lvm comands took
about 15 seconds each.
(although as you say, the lvextend is just a
metadata change...)
> lvextend operation should be very fast operation, this is just a
> metadata change,
> allocating couple of extents to that lv.
>
> Zdenek, how do you suggest to debug slow lvm commands?
>
> See the attached pdf, lvm commands took 15-50 seconds.
>
>>
>> On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
>>>
>>>
>>> On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland <frolland(a)redhat.com>
>>> wrote:
>>> > From the log, we can see that the lvextend command took 18 sec, which
>>> > is
>>> > quite long.
>>>
>>> Fred, can you run repoplot on this log file? it will may explain why
>>> this
>>> lvm
>>> call took 18 seconds.
>>>
>>> Nir
>>>
>>> >
>>> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
>>> > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
>>> > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config '
>>> > devices {
>>> > preferred_names = ["^/dev/mapper/"]
ignore_suspended_devices=1
>>> > write_cache_state=0 disable_after_error_count=3 filter = [
>>> >
'\''a|/dev/mapper/36000eb3a4f1acbc20000000000000043|'\'',
>>> > '\''r|.*|'\''
>>> > ] }
>>> > global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1
>>> > use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } '
>>> > --autobackup
>>> > n --size 6016m
>>> >
>>> >
>>> >
5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
>>> > (cwd None)
>>> > ....
>>> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
>>> > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err>
= '
>>> > WARNING: lvmetad is running but disabled. Restart lvmetad before
>>> > enabling
>>> > it!\n WARNING: This metadata update is NOT backed up\n'; <rc>
= 0
>>> >
>>> >
>>> > The watermark can be configured by the following value:
>>> >
>>> > 'volume_utilization_percent', '50',
>>> > 'Together with volume_utilization_chunk_mb, set the minimal free
'
>>> > 'space before a thin provisioned block volume is extended. Use
'
>>> > 'lower values to extend earlier.')
>>> >
>>> > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek
>>> > <michal.skrivanek(a)redhat.com> wrote:
>>> >>
>>> >>
>>> >> > On 14 Apr 2016, at 09:57, nicolas(a)devels.es wrote:
>>> >> >
>>> >> > Ok, that makes sense, thanks for the insight both Alex and
Fred.
>>> >> > I'm
>>> >> > attaching the VDSM log of the SPM node at the time of the
pause. I
>>> >> > couldn't
>>> >> > find anything that would clearly identify the problem, but
maybe
>>> >> > you'll be
>>> >> > able to.
>>> >>
>>> >> In extreme conditions it will happen. When your storage is slow to
>>> >> respond
>>> >> to extension request, and when your write rate is very high then it
>>> >> may
>>> >> happen, as it is happening to you, that you run out space sooner
than
>>> >> the
>>> >> extension finishes. You can change the watermark value I
guess(right,
>>> >> Fred?), but better would be to plan a bit more ahead and either use
>>> >> preallocated or create thin and then allocate expected size in
>>> >> advance
>>> >> before the operation causing it (typically it only happens during
>>> >> untarring
>>> >> gigabytes of data, or huge database dump/restore)
>>> >> Even then, the VM should always be automatially resumed once the
disk
>>> >> space is allocated
>>> >>
>>> >> Thanks,
>>> >> michal
>>> >>
>>> >> >
>>> >> > Thanks.
>>> >> >
>>> >> > Regards.
>>> >> >
>>> >> > El 2016-04-13 13:09, Fred Rolland escribió:
>>> >> >> Hi,
>>> >> >> Yes, just as Alex explained, if the disk has been created
as thin
>>> >> >> provisioning, the vdsm will extends once a watermark is
reached.
>>> >> >> Usually it should not get to the state the Vm is paused.
>>> >> >> From the log, you can see that the request for extension
has been
>>> >> >> sent
>>> >> >> before the VM got to the No Space Error.
>>> >> >> Later, we can see the VM resuming.
>>> >> >> INFO::2016-04-13
>>> >> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
>>> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting
extension
>>> >> >> for
>>> >> >> volume
>>> >> >> ....
>>> >> >> INFO::2016-04-13
10:52:29,360::vm::3728::virt.vm::(onIOError)
>>> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm
stop
>>> >> >> device
>>> >> >> virtio-disk0 error enospc
>>> >> >> ....
>>> >> >> INFO::2016-04-13
>>> >> >> 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
>>> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running:
onResume
>>> >> >> Note that the extension is done on the SPM host, so it
would be
>>> >> >> interesting to see the vdsm log from the host that was in
SPM role
>>> >> >> at
>>> >> >> this timeframe.
>>> >> >> Regards,
>>> >> >> Fred
>>> >> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow
>>> >> >> <acrow(a)integrafin.co.uk>
>>> >> >> wrote:
>>> >> >>> Hi,
>>> >> >>> If you have set up VM disks as Thin Provisioned, the VM
has to
>>> >> >>> pause when the disk image needs to expand. You
won't see this on
>>> >> >>> VMs
>>> >> >>> with preallocated storage.
>>> >> >>> It's not the SAN that's running out of space,
it's the VM image
>>> >> >>> needing to be expanded incrementally each time.
>>> >> >>> Cheers
>>> >> >>> Alex
>>> >> >>> On 13/04/16 12:04, nicolas(a)devels.es wrote:
>>> >> >>> Hi Fred,
>>> >> >>> This is an iSCSI storage. I'm attaching the VDSM
logs from the
>>> >> >>> host
>>> >> >>> where this machine has been running. Should you need
any further
>>> >> >>> info, don't hesitate to ask.
>>> >> >>> Thanks.
>>> >> >>> Regards.
>>> >> >>> El 2016-04-13 11:54, Fred Rolland escribió:
>>> >> >>> Hi,
>>> >> >>> What kind of storage do you have ? (ISCSI,FC,NFS...)
>>> >> >>> Can you provide the vdsm logs from the host where this
VM runs ?
>>> >> >>> Thanks,
>>> >> >>> Freddy
>>> >> >>> On Wed, Apr 13, 2016 at 1:02 PM,
<nicolas(a)devels.es> wrote:
>>> >> >>> Hi,
>>> >> >>> We're running oVirt 3.6.4.1-1. Lately we're
seeing a bunch of
>>> >> >>> events like these:
>>> >> >>> 2016-04-13 10:52:30,735 INFO
>>> >> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>> >> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
>>> >> >>>
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1])
>>> >> >>> moved
>>> >> >>> from
>>> >> >>> 'Up' --> 'Paused'
>>> >> >>> 2016-04-13 10:52:30,815 INFO
>>> >> >>
>>> >> >>
>>> >> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> >> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f]
Correlation ID:
>>> >> >>> null,
>>> >> >>> Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com
>>> >> >>> [1] [1]
>>> >> >>> has been paused.
>>> >> >>> 2016-04-13 10:52:30,898 ERROR
>>> >> >>
>>> >> >>
>>> >> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> >> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f]
Correlation ID:
>>> >> >>> null,
>>> >> >>> Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com
>>> >> >>> [1] [1]
>>> >> >>> has been paused due to no Storage space error.
>>> >> >>> 2016-04-13 10:52:52,320 WARN
>>> >> >>>
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
>>> >> >>> (org.ovirt.thread.pool-8-thread-38) [] domain
>>> >> >>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01'
in problem. vds:
>>> >> >>> 'host6.domain.com [2] [2]'
>>> >> >>> 2016-04-13 10:52:55,183 INFO
>>> >> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>> >> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
>>> >> >>>
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1])
>>> >> >>> moved
>>> >> >>> from
>>> >> >>> 'Paused' --> 'Up'
>>> >> >>> 2016-04-13 10:52:55,318 INFO
>>> >> >>
>>> >> >>
>>> >> >>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> >> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4]
Correlation ID:
>>> >> >>> null,
>>> >> >>> Call Stack: null, Custom Event ID: -1, Message: VM
vm.domain.com
>>> >> >>> [1] [1]
>>> >> >>> has recovered from paused back to up.
>>> >> >>> The storage domain is far from being full, though (400+
G
>>> >> >>> available
>>> >> >>> right now). Could this be related to this other issue
[1]? If
>>> >> >>> not,
>>> >> >>> how could I debug what's going on?
>>> >> >>> Thanks.
>>> >> >>> [1]:
https://www.mail-archive.com/users@ovirt.org/msg32079.html
>>> >> >>> [3]
>>> >> >>> [3]
>>> >> >>> _______________________________________________
>>> >> >>> Users mailing list
>>> >> >>> Users(a)ovirt.org
>>> >> >>> http://lists.ovirt.org/mailman/listinfo/users [4] [4]
>>> >> >>> Links:
>>> >> >>> ------
>>> >> >>> [1] http://vm.domain.com [1]
>>> >> >>> [2] http://host6.domain.com [2]
>>> >> >>> [3]
https://www.mail-archive.com/users@ovirt.org/msg32079.html
>>> >> >>> [3]
>>> >> >>> [4] http://lists.ovirt.org/mailman/listinfo/users [4]
>>> >> >> _______________________________________________
>>> >> >> Users mailing list
>>> >> >> Users(a)ovirt.org
>>> >> >> http://lists.ovirt.org/mailman/listinfo/users [4]
>>> >> >> --
>>> >> >> This message is intended only for the addressee and may
contain
>>> >> >> confidential information. Unless you are that person, you
may not
>>> >> >> disclose its contents or use it in any way and are
requested to
>>> >> >> delete
>>> >> >> the message along with any attachments and notify us
immediately.
>>> >> >> This email is not intended to, nor should it be taken to,
>>> >> >> constitute
>>> >> >> advice.
>>> >> >> The information provided is correct to our knowledge &
belief and
>>> >> >> must
>>> >> >> not
>>> >> >> be used as a substitute for obtaining tax, regulatory,
investment,
>>> >> >> legal or
>>> >> >> any other appropriate advice.
>>> >> >> "Transact" is operated by Integrated Financial
Arrangements Ltd.
>>> >> >> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608
4900 Fax:
>>> >> >> (020)
>>> >> >> 7608 5300.
>>> >> >> (Registered office: as above; Registered in England and
Wales
>>> >> >> under
>>> >> >> number: 3727592). Authorised and regulated by the
Financial
>>> >> >> Conduct
>>> >> >> Authority (entered on the Financial Services Register; no.
>>> >> >> 190856).
>>> >> >> _______________________________________________
>>> >> >> Users mailing list
>>> >> >> Users(a)ovirt.org
>>> >> >> http://lists.ovirt.org/mailman/listinfo/users [4]
>>> >> >> Links:
>>> >> >> ------
>>> >> >> [1] http://vm.domain.com
>>> >> >> [2] http://host6.domain.com
>>> >> >> [3]
https://www.mail-archive.com/users@ovirt.org/msg32079.html
>>> >> >> [4] http://lists.ovirt.org/mailman/listinfo/users
>>> >> >> _______________________________________________
>>> >> >> Users mailing list
>>> >> >> Users(a)ovirt.org
>>> >> >> http://lists.ovirt.org/mailman/listinfo/users
>>> >> >
<vdsm.log.gz>_______________________________________________
>>> >> > Users mailing list
>>> >> > Users(a)ovirt.org
>>> >> > http://lists.ovirt.org/mailman/listinfo/users
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Users mailing list
>>> > Users(a)ovirt.org
>>> > http://lists.ovirt.org/mailman/listinfo/users
>>> >
>>
>>
>>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
3151
days inactive
3152
days old
13 comments
6 participants
participants (6)
-
Alex Crow -
Colin Coe -
Fred Rolland -
Michal Skrivanek -
nicolas＠devels.es -
Nir Soffer