7:04 a.m.
Hi Folks,
i got a Setup running with the following Specs
4 VM Hosts - CentOS 6.4 - latest Ovirt 3.2 from dreyou
vdsm-xmlrpc-4.10.3-0.36.23.el6.noarch
vdsm-cli-4.10.3-0.36.23.el6.noarch
vdsm-python-4.10.3-0.36.23.el6.x86_64
vdsm-4.10.3-0.36.23.el6.x86_64
qemu-kvm-rhev-tools-0.12.1.2-2.355.el6.5.x86_64
qemu-kvm-rhev-0.12.1.2-2.355.el6.5.x86_64
qemu-img-rhev-0.12.1.2-2.355.el6.5.x86_64
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
Management Node is also running latest 3.2 from dreyou
ovirt-engine-cli-3.2.0.10-1.el6.noarch
ovirt-engine-jbossas711-1-0.x86_64
ovirt-engine-tools-3.2.1-1.41.el6.noarch
ovirt-engine-backend-3.2.1-1.41.el6.noarch
ovirt-engine-sdk-3.2.0.9-1.el6.noarch
ovirt-engine-userportal-3.2.1-1.41.el6.noarch
ovirt-engine-setup-3.2.1-1.41.el6.noarch
ovirt-engine-webadmin-portal-3.2.1-1.41.el6.noarch
ovirt-engine-dbscripts-3.2.1-1.41.el6.noarch
ovirt-engine-3.2.1-1.41.el6.noarch
ovirt-engine-genericapi-3.2.1-1.41.el6.noarch
ovirt-engine-restapi-3.2.1-1.41.el6.noarch
VM are running from a Freebsd 9.1 NFS Server, which works absolutly
flawless until i need to reload the /etc/exports File on the NFS Server.
For this, the NFS Server itself doesnt need to be restarted, just the
mountd Daemon is "Hup´ed".
But after sending a HUP to the mountd, Ovirt immidiatly thinks that there
was a problem with the storage backend and suspends random some VM. Luckily
this VM can be resumed instant without further issues.
The VM Hosts dont show any NFS related errors, so i expect the vdsm or
engine to check the nfs server continous.
The only thing i can find in the vdsm.log of a related host is
-- snip --
Thread-539::DEBUG::2013-07-24
22:29:46,935::resourceManager::830::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-539::DEBUG::2013-07-24
22:29:46,935::resourceManager::864::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-539::DEBUG::2013-07-24
22:29:46,935::task::957::TaskManager.Task::(_decref)
Task=`9332cd24-d899-4226-b0a2-93544ee737b4`::ref 0 aborting False
libvirtEventLoop::INFO::2013-07-24
22:29:55,142::libvirtvm::2509::vm.Vm::(_onAbnormalStop)
vmId=`244f6c8d-bc2b-4669-8f6d-bd957222b946`::abnormal vm stop device
virtio-disk0 error e
other
libvirtEventLoop::DEBUG::2013-07-24
22:29:55,143::libvirtvm::3079::vm.Vm::(_onLibvirtLifecycleEvent)
vmId=`244f6c8d-bc2b-4669-8f6d-bd957222b946`::event Suspended detail 2
opaque No
ne
libvirtEventLoop::INFO::2013-07-24
22:29:55,143::libvirtvm::2509::vm.Vm::(_onAbnormalStop)
vmId=`244f6c8d-bc2b-4669-8f6d-bd957222b946`::abnormal vm stop device
virtio-disk0 error e
other
-- snip --
i am a little bit at a dead end currently, since reloading a nfs servers
export table isnt a unusual task and everything is working like expected.
just ovirt seems way to picky.
is there any possibility to make this check a little bit more tolerant?
i try setting "sd_health_check_delay = 30" in vdsm.conf, but this didnt
change anything.
anyone got an idea how i can get rid of this annoying problem?
Cheers,
Juergen
--
Sent from the Delta quadrant using Borg technology!
7:35 a.m.
Maybe found a workaround on the NFS server side, a option for the mountd
service
-S Tell mountd to suspend/resume execution of the nfsd threads
when-
ever the exports list is being reloaded. This avoids intermit-
tent access errors for clients that do NFS RPCs while the
exports
are being reloaded, but introduces a delay in RPC response
while
the reload is in progress. If mountd crashes while an exports
load is in progress, mountd must be restarted to get the nfsd
threads running again, if this option is used.
so far, i was able to reload the exports list twice, without any random
suspended vm. lets see if this is a real solution or if i just had luck two
times.
but i am still interested in parameters which make the vdsm more tolerant
to short interruptions. instant suspend of a vm after such a short "outage"
is not very nice.
On Wed, Jul 24, 2013 at 11:04 PM, squadra <squadra(a)gmail.com> wrote:
Hi Folks,
i got a Setup running with the following Specs
4 VM Hosts - CentOS 6.4 - latest Ovirt 3.2 from dreyou
vdsm-xmlrpc-4.10.3-0.36.23.el6.noarch
vdsm-cli-4.10.3-0.36.23.el6.noarch
vdsm-python-4.10.3-0.36.23.el6.x86_64
vdsm-4.10.3-0.36.23.el6.x86_64
qemu-kvm-rhev-tools-0.12.1.2-2.355.el6.5.x86_64
qemu-kvm-rhev-0.12.1.2-2.355.el6.5.x86_64
qemu-img-rhev-0.12.1.2-2.355.el6.5.x86_64
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
Management Node is also running latest 3.2 from dreyou
ovirt-engine-cli-3.2.0.10-1.el6.noarch
ovirt-engine-jbossas711-1-0.x86_64
ovirt-engine-tools-3.2.1-1.41.el6.noarch
ovirt-engine-backend-3.2.1-1.41.el6.noarch
ovirt-engine-sdk-3.2.0.9-1.el6.noarch
ovirt-engine-userportal-3.2.1-1.41.el6.noarch
ovirt-engine-setup-3.2.1-1.41.el6.noarch
ovirt-engine-webadmin-portal-3.2.1-1.41.el6.noarch
ovirt-engine-dbscripts-3.2.1-1.41.el6.noarch
ovirt-engine-3.2.1-1.41.el6.noarch
ovirt-engine-genericapi-3.2.1-1.41.el6.noarch
ovirt-engine-restapi-3.2.1-1.41.el6.noarch
VM are running from a Freebsd 9.1 NFS Server, which works absolutly
flawless until i need to reload the /etc/exports File on the NFS Server.
For this, the NFS Server itself doesnt need to be restarted, just the
mountd Daemon is "Hup´ed".
But after sending a HUP to the mountd, Ovirt immidiatly thinks that there
was a problem with the storage backend and suspends random some VM. Luckily
this VM can be resumed instant without further issues.
The VM Hosts dont show any NFS related errors, so i expect the vdsm or
engine to check the nfs server continous.
The only thing i can find in the vdsm.log of a related host is
-- snip --
Thread-539::DEBUG::2013-07-24
22:29:46,935::resourceManager::830::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-539::DEBUG::2013-07-24
22:29:46,935::resourceManager::864::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-539::DEBUG::2013-07-24
22:29:46,935::task::957::TaskManager.Task::(_decref)
Task=`9332cd24-d899-4226-b0a2-93544ee737b4`::ref 0 aborting False
libvirtEventLoop::INFO::2013-07-24
22:29:55,142::libvirtvm::2509::vm.Vm::(_onAbnormalStop)
vmId=`244f6c8d-bc2b-4669-8f6d-bd957222b946`::abnormal vm stop device
virtio-disk0 error e
other
libvirtEventLoop::DEBUG::2013-07-24
22:29:55,143::libvirtvm::3079::vm.Vm::(_onLibvirtLifecycleEvent)
vmId=`244f6c8d-bc2b-4669-8f6d-bd957222b946`::event Suspended detail 2
opaque No
ne
libvirtEventLoop::INFO::2013-07-24
22:29:55,143::libvirtvm::2509::vm.Vm::(_onAbnormalStop)
vmId=`244f6c8d-bc2b-4669-8f6d-bd957222b946`::abnormal vm stop device
virtio-disk0 error e
other
-- snip --
i am a little bit at a dead end currently, since reloading a nfs servers
export table isnt a unusual task and everything is working like expected.
just ovirt seems way to picky.
is there any possibility to make this check a little bit more tolerant?
i try setting "sd_health_check_delay = 30" in vdsm.conf, but this didnt
change anything.
anyone got an idea how i can get rid of this annoying problem?
Cheers,
Juergen
--
Sent from the Delta quadrant using Borg technology!
--
Sent from the Delta quadrant using Borg technology!
9:11 p.m.
--_000_5F9E965F5A80BC468BE5F40576769F092E6EECCEexchange21_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
b25zIDIwMTMtMDctMjQga2xvY2thbiAyMzozNSArMDIwMCBza3JldiBzcXVhZHJhOg0KTWF5YmUg
Zm91bmQgYSB3b3JrYXJvdW5kIG9uIHRoZSBORlMgc2VydmVyIHNpZGUsIGEgb3B0aW9uIGZvciB0
aGUgbW91bnRkIHNlcnZpY2UNCg0KDQoNCg0KICAgICAtUyAgICAgIFRlbGwgbW91bnRkIHRvIHN1
c3BlbmQvcmVzdW1lIGV4ZWN1dGlvbiBvZiB0aGUgbmZzZCB0aHJlYWRzIHdoZW4tDQogICAgICAg
ICAgICAgZXZlciB0aGUgZXhwb3J0cyBsaXN0IGlzIGJlaW5nIHJlbG9hZGVkLiAgVGhpcyBhdm9p
ZHMgaW50ZXJtaXQtDQogICAgICAgICAgICAgdGVudCBhY2Nlc3MgZXJyb3JzIGZvciBjbGllbnRz
IHRoYXQgZG8gTkZTIFJQQ3Mgd2hpbGUgdGhlIGV4cG9ydHMNCiAgICAgICAgICAgICBhcmUgYmVp
bmcgcmVsb2FkZWQsIGJ1dCBpbnRyb2R1Y2VzIGEgZGVsYXkgaW4gUlBDIHJlc3BvbnNlIHdoaWxl
DQogICAgICAgICAgICAgdGhlIHJlbG9hZCBpcyBpbiBwcm9ncmVzcy4gIElmIG1vdW50ZCBjcmFz
aGVzIHdoaWxlIGFuIGV4cG9ydHMNCiAgICAgICAgICAgICBsb2FkIGlzIGluIHByb2dyZXNzLCBt
b3VudGQgbXVzdCBiZSByZXN0YXJ0ZWQgdG8gZ2V0IHRoZSBuZnNkDQogICAgICAgICAgICAgdGhy
ZWFkcyBydW5uaW5nIGFnYWluLCBpZiB0aGlzIG9wdGlvbiBpcyB1c2VkLg0KDQoNCnNvIGZhciwg
aSB3YXMgYWJsZSB0byByZWxvYWQgdGhlIGV4cG9ydHMgbGlzdCB0d2ljZSwgd2l0aG91dCBhbnkg
cmFuZG9tIHN1c3BlbmRlZCB2bS4gbGV0cyBzZWUgaWYgdGhpcyBpcyBhIHJlYWwgc29sdXRpb24g
b3IgaWYgaSBqdXN0IGhhZCBsdWNrIHR3byB0aW1lcy4NCg0KSXQgd291bGQgc2VlbSBhcyBpZiB3
ZSBhcmUgb24gdGhlIHNhbWUgYm9hdDopIEFjdHVhbGx5IEkgaGFkbsK0dCB0aG91Z2h0IGFib3V0
IGl0IGJlZm9yZSwgYnV0IHlvdcK0cmUgcmlnaHQ7IGlzc3VpbmcgYSAic2VydmljZSBtb3VudGQg
cmVsb2FkIiBkb2VzIHBhdXNlIGEgbGFyZ2UgbnVtYmVyIG9mIFZNwrRzLCBmcmlja2luIGFubm95
aW5nIHJlYWxseS4gSSBtZWFuLCB0aGUgTkZTIHNlcnZlciBkb2VzbsK0dCBjYXJlIHdoYXQgb3Ig
d2hvIGl0wrRzIHNlcnZpbmcsIHlvdSBjb3VsZCBiZSBjcmVhdGluZyBhIG5ldyBleHBvcnQgZm9y
IGEgY29tcGxldGVseSBkaWZmZXJlbnQgc3lzdGVtLCBhbmQgbm90IGV2ZW4gaGF2ZSBvVmlydCBp
biBtaW5kIGJlZm9yZSBjdXN0b21lcnMgc3RhcnQgdG8gY2FsbCwgd29uZGVyaW5nIHdoeSB0aGVp
ciBWTcK0cyBoYXZlIHN0b3BwZWQgcmVzcG9uZGluZyE/DQoNCkkgYWN0dWFsbHkgdHJpZWQgdGhh
dCAiLVMiIGJ1dCBpdCBkaWRuwrR0IHdvcmsgZm9yIG1lIGF0IGFsbCwgYW5kIGxvb2tpbmcgYXQg
dGhlIG1hbi1wYWdlIGZvciBtb3VudGQsIHRoZXJlwrRzIG5vIG1lbnRpb24gb2YgaXQgZWl0aGVy
LCBldmVuIHRob3VnaCB3ZSBhcmUgcHJlc3VtYWJseSBydW5uaW5nIHRoZSBzYW1lIHZlcnNpb246
DQojIHVuYW1lIC1yDQo5LjEtUkVMRUFTRQ0KDQpPciBhcmUgeW91IHBlcmhhcHMgdHJhY2tpbmcg
Ii1TVEFCTEUiLCBhbmQgdGhlcmXCtHMgYSBtaW5vciBkaWZmZXJlbmNlIHRoZXJlPw0KDQoNCg0K
YnV0IGkgYW0gc3RpbGwgaW50ZXJlc3RlZCBpbiBwYXJhbWV0ZXJzIHdoaWNoIG1ha2UgdGhlIHZk
c20gbW9yZSB0b2xlcmFudCB0byBzaG9ydCBpbnRlcnJ1cHRpb25zLiBpbnN0YW50IHN1c3BlbmQg
b2YgYSB2bSBhZnRlciBzdWNoIGEgc2hvcnQgIm91dGFnZSIgaXMgbm90IHZlcnkgbmljZS4NCg0K
KzEhDQoNCi9LYXJsaQ0KDQoNCg0KDQoNCg0KDQpPbiBXZWQsIEp1bCAyNCwgMjAxMyBhdCAxMTow
NCBQTSwgc3F1YWRyYSA8c3F1YWRyYUBnbWFpbC5jb208bWFpbHRvOnNxdWFkcmFAZ21haWwuY29t
Pj4gd3JvdGU6DQpIaSBGb2xrcywNCg0KDQppIGdvdCBhIFNldHVwIHJ1bm5pbmcgd2l0aCB0aGUg
Zm9sbG93aW5nIFNwZWNzDQoNCg0KNCBWTSBIb3N0cyAtIENlbnRPUyA2LjQgLSBsYXRlc3QgT3Zp
cnQgMy4yIGZyb20gZHJleW91DQoNCg0KdmRzbS14bWxycGMtNC4xMC4zLTAuMzYuMjMuZWw2Lm5v
YXJjaA0KdmRzbS1jbGktNC4xMC4zLTAuMzYuMjMuZWw2Lm5vYXJjaA0KdmRzbS1weXRob24tNC4x
MC4zLTAuMzYuMjMuZWw2Lng4Nl82NA0KdmRzbS00LjEwLjMtMC4zNi4yMy5lbDYueDg2XzY0DQpx
ZW11LWt2bS1yaGV2LXRvb2xzLTAuMTIuMS4yLTIuMzU1LmVsNi41Lng4Nl82NA0KcWVtdS1rdm0t
cmhldi0wLjEyLjEuMi0yLjM1NS5lbDYuNS54ODZfNjQNCnFlbXUtaW1nLXJoZXYtMC4xMi4xLjIt
Mi4zNTUuZWw2LjUueDg2XzY0DQpncHhlLXJvbXMtcWVtdS0wLjkuNy02LjkuZWw2Lm5vYXJjaA0K
DQoNCk1hbmFnZW1lbnQgTm9kZSBpcyBhbHNvIHJ1bm5pbmcgbGF0ZXN0IDMuMiBmcm9tIGRyZXlv
dQ0KDQoNCm92aXJ0LWVuZ2luZS1jbGktMy4yLjAuMTAtMS5lbDYubm9hcmNoDQpvdmlydC1lbmdp
bmUtamJvc3NhczcxMS0xLTAueDg2XzY0DQpvdmlydC1lbmdpbmUtdG9vbHMtMy4yLjEtMS40MS5l
bDYubm9hcmNoDQpvdmlydC1lbmdpbmUtYmFja2VuZC0zLjIuMS0xLjQxLmVsNi5ub2FyY2gNCm92
aXJ0LWVuZ2luZS1zZGstMy4yLjAuOS0xLmVsNi5ub2FyY2gNCm92aXJ0LWVuZ2luZS11c2VycG9y
dGFsLTMuMi4xLTEuNDEuZWw2Lm5vYXJjaA0Kb3ZpcnQtZW5naW5lLXNldHVwLTMuMi4xLTEuNDEu
ZWw2Lm5vYXJjaA0Kb3ZpcnQtZW5naW5lLXdlYmFkbWluLXBvcnRhbC0zLjIuMS0xLjQxLmVsNi5u
b2FyY2gNCm92aXJ0LWVuZ2luZS1kYnNjcmlwdHMtMy4yLjEtMS40MS5lbDYubm9hcmNoDQpvdmly
dC1lbmdpbmUtMy4yLjEtMS40MS5lbDYubm9hcmNoDQpvdmlydC1lbmdpbmUtZ2VuZXJpY2FwaS0z
LjIuMS0xLjQxLmVsNi5ub2FyY2gNCm92aXJ0LWVuZ2luZS1yZXN0YXBpLTMuMi4xLTEuNDEuZWw2
Lm5vYXJjaA0KDQoNCg0KDQpWTSBhcmUgcnVubmluZyBmcm9tIGEgRnJlZWJzZCA5LjEgTkZTIFNl
cnZlciwgd2hpY2ggd29ya3MgYWJzb2x1dGx5IGZsYXdsZXNzIHVudGlsIGkgbmVlZCB0byByZWxv
YWQgdGhlIC9ldGMvZXhwb3J0cyBGaWxlIG9uIHRoZSBORlMgU2VydmVyLiBGb3IgdGhpcywgdGhl
IE5GUyBTZXJ2ZXIgaXRzZWxmIGRvZXNudCBuZWVkIHRvIGJlIHJlc3RhcnRlZCwganVzdCB0aGUg
bW91bnRkIERhZW1vbiBpcyAiSHVwwrRlZCIuDQoNCg0KQnV0IGFmdGVyIHNlbmRpbmcgYSBIVVAg
dG8gdGhlIG1vdW50ZCwgT3ZpcnQgaW1taWRpYXRseSB0aGlua3MgdGhhdCB0aGVyZSB3YXMgYSBw
cm9ibGVtIHdpdGggdGhlIHN0b3JhZ2UgYmFja2VuZCBhbmQgc3VzcGVuZHMgcmFuZG9tIHNvbWUg
Vk0uIEx1Y2tpbHkgdGhpcyBWTSBjYW4gYmUgcmVzdW1lZCBpbnN0YW50IHdpdGhvdXQgZnVydGhl
ciBpc3N1ZXMuDQoNCg0KVGhlIFZNIEhvc3RzIGRvbnQgc2hvdyBhbnkgTkZTIHJlbGF0ZWQgZXJy
b3JzLCBzbyBpIGV4cGVjdCB0aGUgdmRzbSBvciBlbmdpbmUgdG8gY2hlY2sgdGhlIG5mcyBzZXJ2
ZXIgY29udGlub3VzLg0KDQoNClRoZSBvbmx5IHRoaW5nIGkgY2FuIGZpbmQgaW4gdGhlIHZkc20u
bG9nIG9mIGEgcmVsYXRlZCBob3N0IGlzDQoNCg0KLS0gc25pcCAtLQ0KDQoNClRocmVhZC01Mzk6
OkRFQlVHOjoyMDEzLTA3LTI0IDIyOjI5OjQ2LDkzNTo6cmVzb3VyY2VNYW5hZ2VyOjo4MzA6OlJl
c291cmNlTWFuYWdlci5Pd25lcjo6KHJlbGVhc2VBbGwpIE93bmVyLnJlbGVhc2VBbGwgcmVxdWVz
dHMge30gcmVzb3VyY2VzIHt9DQpUaHJlYWQtNTM5OjpERUJVRzo6MjAxMy0wNy0yNCAyMjoyOTo0
Niw5MzU6OnJlc291cmNlTWFuYWdlcjo6ODY0OjpSZXNvdXJjZU1hbmFnZXIuT3duZXI6OihjYW5j
ZWxBbGwpIE93bmVyLmNhbmNlbEFsbCByZXF1ZXN0cyB7fQ0KVGhyZWFkLTUzOTo6REVCVUc6OjIw
MTMtMDctMjQgMjI6Mjk6NDYsOTM1Ojp0YXNrOjo5NTc6OlRhc2tNYW5hZ2VyLlRhc2s6OihfZGVj
cmVmKSBUYXNrPWA5MzMyY2QyNC1kODk5LTQyMjYtYjBhMi05MzU0NGVlNzM3YjRgOjpyZWYgMCBh
Ym9ydGluZyBGYWxzZQ0KbGlidmlydEV2ZW50TG9vcDo6SU5GTzo6MjAxMy0wNy0yNCAyMjoyOTo1
NSwxNDI6OmxpYnZpcnR2bTo6MjUwOTo6dm0uVm06Oihfb25BYm5vcm1hbFN0b3ApIHZtSWQ9YDI0
NGY2YzhkLWJjMmItNDY2OS04ZjZkLWJkOTU3MjIyYjk0NmA6OmFibm9ybWFsIHZtIHN0b3AgZGV2
aWNlIHZpcnRpby1kaXNrMCBlcnJvciBlDQpvdGhlcg0KbGlidmlydEV2ZW50TG9vcDo6REVCVUc6
OjIwMTMtMDctMjQgMjI6Mjk6NTUsMTQzOjpsaWJ2aXJ0dm06OjMwNzk6OnZtLlZtOjooX29uTGli
dmlydExpZmVjeWNsZUV2ZW50KSB2bUlkPWAyNDRmNmM4ZC1iYzJiLTQ2NjktOGY2ZC1iZDk1NzIy
MmI5NDZgOjpldmVudCBTdXNwZW5kZWQgZGV0YWlsIDIgb3BhcXVlIE5vDQpuZQ0KbGlidmlydEV2
ZW50TG9vcDo6SU5GTzo6MjAxMy0wNy0yNCAyMjoyOTo1NSwxNDM6OmxpYnZpcnR2bTo6MjUwOTo6
dm0uVm06Oihfb25BYm5vcm1hbFN0b3ApIHZtSWQ9YDI0NGY2YzhkLWJjMmItNDY2OS04ZjZkLWJk
OTU3MjIyYjk0NmA6OmFibm9ybWFsIHZtIHN0b3AgZGV2aWNlIHZpcnRpby1kaXNrMCBlcnJvciBl
DQpvdGhlcg0KDQoNCg0KDQotLSBzbmlwIC0tDQoNCg0KaSBhbSBhIGxpdHRsZSBiaXQgYXQgYSBk
ZWFkIGVuZCBjdXJyZW50bHksIHNpbmNlIHJlbG9hZGluZyBhIG5mcyBzZXJ2ZXJzIGV4cG9ydCB0
YWJsZSBpc250IGEgdW51c3VhbCB0YXNrIGFuZCBldmVyeXRoaW5nIGlzIHdvcmtpbmcgbGlrZSBl
eHBlY3RlZC4ganVzdCBvdmlydCBzZWVtcyB3YXkgdG8gcGlja3kuDQoNCg0KaXMgdGhlcmUgYW55
IHBvc3NpYmlsaXR5IHRvIG1ha2UgdGhpcyBjaGVjayBhIGxpdHRsZSBiaXQgbW9yZSB0b2xlcmFu
dD8NCg0KDQppIHRyeSBzZXR0aW5nICJzZF9oZWFsdGhfY2hlY2tfZGVsYXkgPSAzMCIgaW4gdmRz
bS5jb25mLCBidXQgdGhpcyBkaWRudCBjaGFuZ2UgYW55dGhpbmcuDQoNCg0KYW55b25lIGdvdCBh
biBpZGVhIGhvdyBpIGNhbiBnZXQgcmlkIG9mIHRoaXMgYW5ub3lpbmcgcHJvYmxlbT8NCg0KDQpD
aGVlcnMsDQoNCg0KSnVlcmdlbg0KDQoNCg0KDQotLQ0KDQoNClNlbnQgZnJvbSB0aGUgRGVsdGEg
cXVhZHJhbnQgdXNpbmcgQm9yZyB0ZWNobm9sb2d5IQ0KDQoNCg0KDQoNCi0tDQoNCg0KU2VudCBm
cm9tIHRoZSBEZWx0YSBxdWFkcmFudCB1c2luZyBCb3JnIHRlY2hub2xvZ3khDQoNCg0KLS0NCg0K
TWVkIFbDpG5saWdhIEjDpGxzbmluZ2FyDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQpLYXJsaSBT
asO2YmVyZw0KU3dlZGlzaCBVbml2ZXJzaXR5IG9mIEFncmljdWx0dXJhbCBTY2llbmNlcw0KQm94
IDcwNzkgKFZpc2l0aW5nIEFkZHJlc3MgS3JvbsOlc3bDpGdlbiA4KQ0KUy03NTAgMDcgVXBwc2Fs
YSwgU3dlZGVuDQpQaG9uZTogICs0Ni0oMCkxOC02NyAxNSA2Ng0Ka2FybGkuc2pvYmVyZ0BzbHUu
c2U8bWFpbHRvOmthcmxpLnNqb2JlcmdAYWRtLnNsdS5zZT4NCg==
--_000_5F9E965F5A80BC468BE5F40576769F092E6EECCEexchange21_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64
PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMCBUUkFOU0lUSU9OQUwv
L0VOIj4NCjxodG1sPg0KPGhlYWQ+DQo8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNv
bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+DQo8bWV0YSBuYW1lPSJHRU5FUkFUT1Ii
IGNvbnRlbnQ9Ikd0a0hUTUwvNC40LjQiPg0KPC9oZWFkPg0KPGJvZHk+DQpvbnMgMjAxMy0wNy0y
NCBrbG9ja2FuIDIzOjM1ICYjNDM7MDIwMCBza3JldiBzcXVhZHJhOg0KPGJsb2NrcXVvdGUgdHlw
ZT0iQ0lURSI+TWF5YmUgZm91bmQgYSB3b3JrYXJvdW5kIG9uIHRoZSBORlMgc2VydmVyIHNpZGUs
IGEgb3B0aW9uIGZvciB0aGUgbW91bnRkIHNlcnZpY2UNCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1
b3RlIHR5cGU9IkNJVEUiPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5
cGU9IkNJVEUiPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJ
VEUiPiZuYnNwOyAmbmJzcDsgJm5ic3A7LVMgJm5ic3A7ICZuYnNwOyAmbmJzcDtUZWxsIG1vdW50
ZCB0byBzdXNwZW5kL3Jlc3VtZSBleGVjdXRpb24gb2YgdGhlIG5mc2QgdGhyZWFkcyB3aGVuLQ0K
PC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+Jm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ZXZlciB0aGUgZXhwb3J0cyBsaXN0IGlzIGJl
aW5nIHJlbG9hZGVkLiAmbmJzcDtUaGlzIGF2b2lkcyBpbnRlcm1pdC0NCjwvYmxvY2txdW90ZT4N
CjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwO3RlbnQgYWNjZXNzIGVycm9ycyBmb3IgY2xpZW50cyB0aGF0IGRvIE5G
UyBSUENzIHdoaWxlIHRoZSBleHBvcnRzDQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBl
PSJDSVRFIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDth
cmUgYmVpbmcgcmVsb2FkZWQsIGJ1dCBpbnRyb2R1Y2VzIGEgZGVsYXkgaW4gUlBDIHJlc3BvbnNl
IHdoaWxlDQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4mbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDt0aGUgcmVsb2FkIGlzIGluIHBy
b2dyZXNzLiAmbmJzcDtJZiBtb3VudGQgY3Jhc2hlcyB3aGlsZSBhbiBleHBvcnRzDQo8L2Jsb2Nr
cXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtsb2FkIGlzIGluIHByb2dyZXNzLCBtb3VudGQgbXVzdCBi
ZSByZXN0YXJ0ZWQgdG8gZ2V0IHRoZSBuZnNkDQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0
eXBlPSJDSVRFIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDt0aHJlYWRzIHJ1bm5pbmcgYWdhaW4sIGlmIHRoaXMgb3B0aW9uIGlzIHVzZWQuDQo8L2Jsb2Nr
cXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj48YnI+DQo8YnI+DQo8L2Jsb2NrcXVvdGU+
DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj5zbyBmYXIsIGkgd2FzIGFibGUgdG8gcmVsb2FkIHRo
ZSBleHBvcnRzIGxpc3QgdHdpY2UsIHdpdGhvdXQgYW55IHJhbmRvbSBzdXNwZW5kZWQgdm0uIGxl
dHMgc2VlIGlmIHRoaXMgaXMgYSByZWFsIHNvbHV0aW9uIG9yIGlmIGkganVzdCBoYWQgbHVjayB0
d28gdGltZXMuPGJyPg0KPC9ibG9ja3F1b3RlPg0KPGJyPg0KSXQgd291bGQgc2VlbSBhcyBpZiB3
ZSBhcmUgb24gdGhlIHNhbWUgYm9hdDopIEFjdHVhbGx5IEkgaGFkbsK0dCB0aG91Z2h0IGFib3V0
IGl0IGJlZm9yZSwgYnV0IHlvdcK0cmUgcmlnaHQ7IGlzc3VpbmcgYSAmcXVvdDtzZXJ2aWNlIG1v
dW50ZCByZWxvYWQmcXVvdDsgZG9lcyBwYXVzZSBhIGxhcmdlIG51bWJlciBvZiBWTcK0cywgZnJp
Y2tpbiBhbm5veWluZyByZWFsbHkuIEkgbWVhbiwgdGhlIE5GUyBzZXJ2ZXIgZG9lc27CtHQgY2Fy
ZSB3aGF0IG9yIHdobyBpdMK0cyBzZXJ2aW5nLA0KIHlvdSBjb3VsZCBiZSBjcmVhdGluZyBhIG5l
dyBleHBvcnQgZm9yIGEgY29tcGxldGVseSBkaWZmZXJlbnQgc3lzdGVtLCBhbmQgbm90IGV2ZW4g
aGF2ZSBvVmlydCBpbiBtaW5kIGJlZm9yZSBjdXN0b21lcnMgc3RhcnQgdG8gY2FsbCwgd29uZGVy
aW5nIHdoeSB0aGVpciBWTcK0cyBoYXZlIHN0b3BwZWQgcmVzcG9uZGluZyE/PGJyPg0KPGJyPg0K
SSBhY3R1YWxseSB0cmllZCB0aGF0ICZxdW90Oy1TJnF1b3Q7IGJ1dCBpdCBkaWRuwrR0IHdvcmsg
Zm9yIG1lIGF0IGFsbCwgYW5kIGxvb2tpbmcgYXQgdGhlIG1hbi1wYWdlIGZvciBtb3VudGQsIHRo
ZXJlwrRzIG5vIG1lbnRpb24gb2YgaXQgZWl0aGVyLCBldmVuIHRob3VnaCB3ZSBhcmUgcHJlc3Vt
YWJseSBydW5uaW5nIHRoZSBzYW1lIHZlcnNpb246PGJyPg0KIyB1bmFtZSAtcjxicj4NCjkuMS1S
RUxFQVNFPGJyPg0KPGJyPg0KT3IgYXJlIHlvdSBwZXJoYXBzIHRyYWNraW5nICZxdW90Oy1TVEFC
TEUmcXVvdDssIGFuZCB0aGVyZcK0cyBhIG1pbm9yIGRpZmZlcmVuY2UgdGhlcmU/PGJyPg0KPGJy
Pg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+PGJyPg0KPGJyPg0KPC9ibG9ja3F1b3RlPg0KPGJs
b2NrcXVvdGUgdHlwZT0iQ0lURSI+YnV0IGkgYW0gc3RpbGwgaW50ZXJlc3RlZCBpbiBwYXJhbWV0
ZXJzIHdoaWNoIG1ha2UgdGhlIHZkc20gbW9yZSB0b2xlcmFudCB0byBzaG9ydCBpbnRlcnJ1cHRp
b25zLiBpbnN0YW50IHN1c3BlbmQgb2YgYSB2bSBhZnRlciBzdWNoIGEgc2hvcnQgJnF1b3Q7b3V0
YWdlJnF1b3Q7IGlzIG5vdCB2ZXJ5IG5pY2UuPGJyPg0KPC9ibG9ja3F1b3RlPg0KPGJyPg0KJiM0
MzsxITxicj4NCjxicj4NCi9LYXJsaTxicj4NCjxicj4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUi
Pjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPjxicj4N
Cjxicj4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPjxicj4NCjxicj4N
CjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPk9uIFdlZCwgSnVsIDI0LCAy
MDEzIGF0IDExOjA0IFBNLCBzcXVhZHJhICZsdDs8YSBocmVmPSJtYWlsdG86c3F1YWRyYUBnbWFp
bC5jb20iPnNxdWFkcmFAZ21haWwuY29tPC9hPiZndDsgd3JvdGU6DQo8L2Jsb2NrcXVvdGU+DQo8
YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPkhpIEZvbGtzLCA8L2Jsb2NrcXVv
dGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3Rl
Pjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5
cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+aSBnb3QgYSBTZXR1cCBydW5uaW5nIHdpdGggdGhlIGZv
bGxvd2luZyBTcGVjcyA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0
eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjwvYmxv
Y2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+NCBWTSBIb3N0
cyAtIENlbnRPUyA2LjQgLSBsYXRlc3QgT3ZpcnQgMy4yIGZyb20gZHJleW91IDwvYmxvY2txdW90
ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+
PGJyPg0KPGJyPg0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlw
ZT0iQ0lURSI+DQo8YmxvY2txdW90ZT52ZHNtLXhtbHJwYy00LjEwLjMtMC4zNi4yMy5lbDYubm9h
cmNoIDwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUi
Pg0KPGJsb2NrcXVvdGU+dmRzbS1jbGktNC4xMC4zLTAuMzYuMjMuZWw2Lm5vYXJjaCA8L2Jsb2Nr
cXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1
b3RlPnZkc20tcHl0aG9uLTQuMTAuMy0wLjM2LjIzLmVsNi54ODZfNjQgPC9ibG9ja3F1b3RlPg0K
PC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90ZT52ZHNt
LTQuMTAuMy0wLjM2LjIzLmVsNi54ODZfNjQgPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0K
PGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90ZT5xZW11LWt2bS1yaGV2LXRvb2xz
LTAuMTIuMS4yLTIuMzU1LmVsNi41Lng4Nl82NCA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+
DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPnFlbXUta3ZtLXJoZXYtMC4x
Mi4xLjItMi4zNTUuZWw2LjUueDg2XzY0IDwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxi
bG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+cWVtdS1pbWctcmhldi0wLjEyLjEu
Mi0yLjM1NS5lbDYuNS54ODZfNjQgPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2Nr
cXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90ZT5ncHhlLXJvbXMtcWVtdS0wLjkuNy02Ljku
ZWw2Lm5vYXJjaCA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBl
PSJDSVRFIj4NCjxibG9ja3F1b3RlPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjwvYmxvY2tx
dW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+TWFuYWdlbWVudCBO
b2RlIGlzIGFsc28gcnVubmluZyBsYXRlc3QgMy4yIGZyb20gZHJleW91IDwvYmxvY2txdW90ZT4N
CjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+PGJy
Pg0KPGJyPg0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0i
Q0lURSI+DQo8YmxvY2txdW90ZT5vdmlydC1lbmdpbmUtY2xpLTMuMi4wLjEwLTEuZWw2Lm5vYXJj
aCA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4N
CjxibG9ja3F1b3RlPm92aXJ0LWVuZ2luZS1qYm9zc2FzNzExLTEtMC54ODZfNjQgPC9ibG9ja3F1
b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90
ZT5vdmlydC1lbmdpbmUtdG9vbHMtMy4yLjEtMS40MS5lbDYubm9hcmNoIDwvYmxvY2txdW90ZT4N
CjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+b3Zp
cnQtZW5naW5lLWJhY2tlbmQtMy4yLjEtMS40MS5lbDYubm9hcmNoIDwvYmxvY2txdW90ZT4NCjwv
YmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+b3ZpcnQt
ZW5naW5lLXNkay0zLjIuMC45LTEuZWw2Lm5vYXJjaCA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVv
dGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPm92aXJ0LWVuZ2luZS11
c2VycG9ydGFsLTMuMi4xLTEuNDEuZWw2Lm5vYXJjaCA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVv
dGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPm92aXJ0LWVuZ2luZS1z
ZXR1cC0zLjIuMS0xLjQxLmVsNi5ub2FyY2ggPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0K
PGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90ZT5vdmlydC1lbmdpbmUtd2ViYWRt
aW4tcG9ydGFsLTMuMi4xLTEuNDEuZWw2Lm5vYXJjaCA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVv
dGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPm92aXJ0LWVuZ2luZS1k
YnNjcmlwdHMtMy4yLjEtMS40MS5lbDYubm9hcmNoIDwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90
ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+b3ZpcnQtZW5naW5lLTMu
Mi4xLTEuNDEuZWw2Lm5vYXJjaCA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2tx
dW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPm92aXJ0LWVuZ2luZS1nZW5lcmljYXBpLTMu
Mi4xLTEuNDEuZWw2Lm5vYXJjaCA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2tx
dW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPm92aXJ0LWVuZ2luZS1yZXN0YXBpLTMuMi4x
LTEuNDEuZWw2Lm5vYXJjaCA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90
ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjwv
YmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+PGJyPg0K
PGJyPg0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lU
RSI+DQo8YmxvY2txdW90ZT5WTSBhcmUgcnVubmluZyBmcm9tIGEgRnJlZWJzZCA5LjEgTkZTIFNl
cnZlciwgd2hpY2ggd29ya3MgYWJzb2x1dGx5IGZsYXdsZXNzIHVudGlsIGkgbmVlZCB0byByZWxv
YWQgdGhlIC9ldGMvZXhwb3J0cyBGaWxlIG9uIHRoZSBORlMgU2VydmVyLiBGb3IgdGhpcywgdGhl
IE5GUyBTZXJ2ZXIgaXRzZWxmIGRvZXNudCBuZWVkIHRvIGJlIHJlc3RhcnRlZCwganVzdCB0aGUg
bW91bnRkIERhZW1vbiBpcyAmcXVvdDtIdXDCtGVkJnF1b3Q7LiZuYnNwOw0KPC9ibG9ja3F1b3Rl
Pg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90ZT48
YnI+DQo8YnI+DQo8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBl
PSJDSVRFIj4NCjxibG9ja3F1b3RlPkJ1dCBhZnRlciBzZW5kaW5nIGEgSFVQIHRvIHRoZSBtb3Vu
dGQsIE92aXJ0IGltbWlkaWF0bHkgdGhpbmtzIHRoYXQgdGhlcmUgd2FzIGEgcHJvYmxlbSB3aXRo
IHRoZSBzdG9yYWdlIGJhY2tlbmQgYW5kIHN1c3BlbmRzIHJhbmRvbSBzb21lIFZNLiBMdWNraWx5
IHRoaXMgVk0gY2FuIGJlIHJlc3VtZWQgaW5zdGFudCB3aXRob3V0IGZ1cnRoZXIgaXNzdWVzLiZu
YnNwOw0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lU
RSI+DQo8YmxvY2txdW90ZT48YnI+DQo8YnI+DQo8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+
DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPlRoZSBWTSBIb3N0cyBkb250
IHNob3cgYW55IE5GUyByZWxhdGVkIGVycm9ycywgc28gaSBleHBlY3QgdGhlIHZkc20gb3IgZW5n
aW5lIHRvIGNoZWNrIHRoZSBuZnMgc2VydmVyIGNvbnRpbm91cy4mbmJzcDsNCjwvYmxvY2txdW90
ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+
PGJyPg0KPGJyPg0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlw
ZT0iQ0lURSI+DQo8YmxvY2txdW90ZT5UaGUgb25seSB0aGluZyBpIGNhbiBmaW5kIGluIHRoZSB2
ZHNtLmxvZyBvZiBhIHJlbGF0ZWQgaG9zdCBpcyA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+
DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPjxicj4NCjxicj4NCjwvYmxv
Y2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2Nr
cXVvdGU+LS0gc25pcCAtLSA8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90
ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjwv
YmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+VGhyZWFk
LTUzOTo6REVCVUc6OjIwMTMtMDctMjQgMjI6Mjk6NDYsOTM1OjpyZXNvdXJjZU1hbmFnZXI6Ojgz
MDo6UmVzb3VyY2VNYW5hZ2VyLk93bmVyOjoocmVsZWFzZUFsbCkgT3duZXIucmVsZWFzZUFsbCBy
ZXF1ZXN0cyB7fSByZXNvdXJjZXMge30NCjwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxi
bG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+VGhyZWFkLTUzOTo6REVCVUc6OjIw
MTMtMDctMjQgMjI6Mjk6NDYsOTM1OjpyZXNvdXJjZU1hbmFnZXI6Ojg2NDo6UmVzb3VyY2VNYW5h
Z2VyLk93bmVyOjooY2FuY2VsQWxsKSBPd25lci5jYW5jZWxBbGwgcmVxdWVzdHMge30NCjwvYmxv
Y2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2Nr
cXVvdGU+VGhyZWFkLTUzOTo6REVCVUc6OjIwMTMtMDctMjQgMjI6Mjk6NDYsOTM1Ojp0YXNrOjo5
NTc6OlRhc2tNYW5hZ2VyLlRhc2s6OihfZGVjcmVmKSBUYXNrPWA5MzMyY2QyNC1kODk5LTQyMjYt
YjBhMi05MzU0NGVlNzM3YjRgOjpyZWYgMCBhYm9ydGluZyBGYWxzZQ0KPC9ibG9ja3F1b3RlPg0K
PC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90ZT5saWJ2
aXJ0RXZlbnRMb29wOjpJTkZPOjoyMDEzLTA3LTI0IDIyOjI5OjU1LDE0Mjo6bGlidmlydHZtOjoy
NTA5Ojp2bS5WbTo6KF9vbkFibm9ybWFsU3RvcCkgdm1JZD1gMjQ0ZjZjOGQtYmMyYi00NjY5LThm
NmQtYmQ5NTcyMjJiOTQ2YDo6YWJub3JtYWwgdm0gc3RvcCBkZXZpY2UgdmlydGlvLWRpc2swIGVy
cm9yIGUNCjwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJ
VEUiPg0KPGJsb2NrcXVvdGU+b3RoZXIgPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJs
b2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90ZT5saWJ2aXJ0RXZlbnRMb29wOjpERUJV
Rzo6MjAxMy0wNy0yNCAyMjoyOTo1NSwxNDM6OmxpYnZpcnR2bTo6MzA3OTo6dm0uVm06Oihfb25M
aWJ2aXJ0TGlmZWN5Y2xlRXZlbnQpIHZtSWQ9YDI0NGY2YzhkLWJjMmItNDY2OS04ZjZkLWJkOTU3
MjIyYjk0NmA6OmV2ZW50IFN1c3BlbmRlZCBkZXRhaWwgMiBvcGFxdWUgTm8NCjwvYmxvY2txdW90
ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+
bmUgPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+
DQo8YmxvY2txdW90ZT5saWJ2aXJ0RXZlbnRMb29wOjpJTkZPOjoyMDEzLTA3LTI0IDIyOjI5OjU1
LDE0Mzo6bGlidmlydHZtOjoyNTA5Ojp2bS5WbTo6KF9vbkFibm9ybWFsU3RvcCkgdm1JZD1gMjQ0
ZjZjOGQtYmMyYi00NjY5LThmNmQtYmQ5NTcyMjJiOTQ2YDo6YWJub3JtYWwgdm0gc3RvcCBkZXZp
Y2UgdmlydGlvLWRpc2swIGVycm9yIGUNCjwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxi
bG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+b3RoZXIgPC9ibG9ja3F1b3RlPg0K
PC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90ZT48YnI+
DQo8YnI+DQo8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJD
SVRFIj4NCjxibG9ja3F1b3RlPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90
ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+LS0gc25pcCAtLSA8L2Js
b2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9j
a3F1b3RlPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1
b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+aSBhbSBhIGxpdHRsZSBiaXQgYXQgYSBkZWFk
IGVuZCBjdXJyZW50bHksIHNpbmNlIHJlbG9hZGluZyBhIG5mcyBzZXJ2ZXJzIGV4cG9ydCB0YWJs
ZSBpc250IGEgdW51c3VhbCB0YXNrIGFuZCBldmVyeXRoaW5nIGlzIHdvcmtpbmcgbGlrZSBleHBl
Y3RlZC4ganVzdCBvdmlydCBzZWVtcyB3YXkgdG8gcGlja3kuJm5ic3A7DQo8L2Jsb2NrcXVvdGU+
DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPjxi
cj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9
IkNJVEUiPg0KPGJsb2NrcXVvdGU+aXMgdGhlcmUgYW55IHBvc3NpYmlsaXR5IHRvIG1ha2UgdGhp
cyBjaGVjayBhIGxpdHRsZSBiaXQgbW9yZSB0b2xlcmFudD8NCjwvYmxvY2txdW90ZT4NCjwvYmxv
Y2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+PGJyPg0KPGJy
Pg0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+
DQo8YmxvY2txdW90ZT5pIHRyeSBzZXR0aW5nICZxdW90O3NkX2hlYWx0aF9jaGVja19kZWxheSA9
IDMwJnF1b3Q7IGluIHZkc20uY29uZiwgYnV0IHRoaXMgZGlkbnQgY2hhbmdlIGFueXRoaW5nLg0K
PC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8
YmxvY2txdW90ZT48YnI+DQo8YnI+DQo8L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8Ymxv
Y2txdW90ZSB0eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPmFueW9uZSBnb3QgYW4gaWRlYSBob3cg
aSBjYW4gZ2V0IHJpZCBvZiB0aGlzIGFubm95aW5nIHByb2JsZW0/IDwvYmxvY2txdW90ZT4NCjwv
YmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+PGJyPg0K
PGJyPg0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lU
RSI+DQo8YmxvY2txdW90ZT5DaGVlcnMsIDwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxi
bG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+PGJyPg0KPGJyPg0KPC9ibG9ja3F1
b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90
ZT5KdWVyZ2VuIDwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9
IkNJVEUiPg0KPGJsb2NrcXVvdGU+PGJyPg0KPGJyPg0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1
b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90ZT48YnI+DQo8YnI+DQo8
L2Jsb2NrcXVvdGU+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxi
bG9ja3F1b3RlPjxmb250IGNvbG9yPSIjODg4ODg4Ij4tLSA8L2ZvbnQ+DQo8cHJlPgo8Zm9udCBj
b2xvcj0iIzg4ODg4OCI+U2VudCBmcm9tIHRoZSBEZWx0YSBxdWFkcmFudCB1c2luZyBCb3JnIHRl
Y2hub2xvZ3khPC9mb250Pgo8L3ByZT4NCjwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxi
bG9ja3F1b3RlIHR5cGU9IkNJVEUiPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1
b3RlIHR5cGU9IkNJVEUiPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5
cGU9IkNJVEUiPi0tDQo8cHJlPgpTZW50IGZyb20gdGhlIERlbHRhIHF1YWRyYW50IHVzaW5nIEJv
cmcgdGVjaG5vbG9neSEKPC9wcmU+DQo8L2Jsb2NrcXVvdGU+DQo8YnI+DQo8dGFibGUgY2VsbHNw
YWNpbmc9IjAiIGNlbGxwYWRkaW5nPSIwIiB3aWR0aD0iMTAwJSI+DQo8dGJvZHk+DQo8dHI+DQo8
dGQ+LS0gPGJyPg0KPGJyPg0KTWVkIFbDpG5saWdhIEjDpGxzbmluZ2FyPGJyPg0KLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLTxicj4NCkthcmxpIFNqw7ZiZXJnPGJyPg0KU3dlZGlzaCBVbml2ZXJzaXR5
IG9mIEFncmljdWx0dXJhbCBTY2llbmNlczxicj4NCkJveCA3MDc5IChWaXNpdGluZyBBZGRyZXNz
IEtyb27DpXN2w6RnZW4gOCk8YnI+DQpTLTc1MCAwNyBVcHBzYWxhLCBTd2VkZW48YnI+DQpQaG9u
ZTogJm5ic3A7JiM0Mzs0Ni0oMCkxOC02NyAxNSA2Njxicj4NCjxhIGhyZWY9Im1haWx0bzprYXJs
aS5zam9iZXJnQGFkbS5zbHUuc2UiPmthcmxpLnNqb2JlcmdAc2x1LnNlPC9hPiA8L3RkPg0KPC90
cj4NCjwvdGJvZHk+DQo8L3RhYmxlPg0KPC9ib2R5Pg0KPC9odG1sPg0K
--_000_5F9E965F5A80BC468BE5F40576769F092E6EECCEexchange21_--
9:26 p.m.
Hi Karli,
i already thought that i am the only one with that combination ;)
On Mon, Jul 29, 2013 at 1:11 PM, Karli Sjöberg <Karli.Sjoberg(a)slu.se> wrote:
**
ons 2013-07-24 klockan 23:35 +0200 skrev squadra:
Maybe found a workaround on the NFS server side, a option for the mountd
service
-S Tell mountd to suspend/resume execution of the nfsd threads
when-
ever the exports list is being reloaded. This avoids
intermit-
tent access errors for clients that do NFS RPCs while the
exports
are being reloaded, but introduces a delay in RPC response
while
the reload is in progress. If mountd crashes while an
exports
load is in progress, mountd must be restarted to get the nfsd
threads running again, if this option is used.
so far, i was able to reload the exports list twice, without any random
suspended vm. lets see if this is a real solution or if i just had luck two
times.
It would seem as if we are on the same boat:) Actually I hadn´t thought
about it before, but you´re right; issuing a "service mountd reload" does
pause a large number of VM´s, frickin annoying really. I mean, the NFS
server doesn´t care what or who it´s serving, you could be creating a new
export for a completely different system, and not even have oVirt in mind
before customers start to call, wondering why their VM´s have stopped
responding!?
exactly the same i have
I actually tried that "-S" but it didn´t work for me at
all, and looking
at the man-page for mountd, there´s no mention of it either, even though we
are presumably running the same version:
# uname -r
9.1-RELEASE
Or are you perhaps tracking "-STABLE", and there´s a minor difference
there?
i am tracking -STABLE, but the Man Page of "mountd" on a 9.1 Stable
(Snapshot Release) also shows -S Parameter
9.1-STABLE FreeBSD 9.1-STABLE #0: Sun Jul 7 10:53:46 UTC 2013
root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
9.2-PRERELEASE FreeBSD 9.2-PRERELEASE #6: Thu Jul 18 02:41:57 CEST 2013
root(a)filer1.intern.
´
both systems provide -S for the mountd, and so far i didnt have any more
problems. lets see if this keeps going good.
but i am still interested in parameters which make the vdsm more
tolerant to short interruptions. instant suspend of a vm after such a short
"outage" is not very nice.
+1!
/Karli
Cheers,
Juergen
--
Sent from the Delta quadrant using Borg technology!
9:46 p.m.
--_000_5F9E965F5A80BC468BE5F40576769F092E6EED87exchange21_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
bcOlbiAyMDEzLTA3LTI5IGtsb2NrYW4gMTM6MjYgKzAyMDAgc2tyZXYgc3F1YWRyYToNCkhpIEth
cmxpLA0KDQoNCmkgYWxyZWFkeSB0aG91Z2h0IHRoYXQgaSBhbSB0aGUgb25seSBvbmUgd2l0aCB0
aGF0IGNvbWJpbmF0aW9uIDspDQoNCldlbGwsIEkgaGFwcGVuIHRvIGJlIHVzaW5nIEZlZG9yYSBh
cyBlbmdpbmUvaG9zdHMsIGJ1dCB3aGVuIGl0IGNvbWVzIHRvIHRoZSBORlMtc2VydmVyLCB3aHkg
c2V0dGxlIGZvciBhbnl0aGluZyBsZXNzLCByaWdodD86KSBJIGltYWdpbmUgeW91wrRyZSBpbiBp
dCBmb3IgdGhlIHNhbWUgcmVhc29uIGFzIG1lIHRvbzsgInRoZSBsYXN0IHdvcmQgaW4gZmlsZXN5
c3RlbXMiLi4uDQoNCg0KDQpPbiBNb24sIEp1bCAyOSwgMjAxMyBhdCAxOjExIFBNLCBLYXJsaSBT
asO2YmVyZyA8S2FybGkuU2pvYmVyZ0BzbHUuc2U8bWFpbHRvOkthcmxpLlNqb2JlcmdAc2x1LnNl
Pj4gd3JvdGU6DQpvbnMgMjAxMy0wNy0yNCBrbG9ja2FuIDIzOjM1ICswMjAwIHNrcmV2IHNxdWFk
cmE6DQpNYXliZSBmb3VuZCBhIHdvcmthcm91bmQgb24gdGhlIE5GUyBzZXJ2ZXIgc2lkZSwgYSBv
cHRpb24gZm9yIHRoZSBtb3VudGQgc2VydmljZQ0KDQoNCg0KDQogICAgIC1TICAgICAgVGVsbCBt
b3VudGQgdG8gc3VzcGVuZC9yZXN1bWUgZXhlY3V0aW9uIG9mIHRoZSBuZnNkIHRocmVhZHMgd2hl
bi0NCiAgICAgICAgICAgICBldmVyIHRoZSBleHBvcnRzIGxpc3QgaXMgYmVpbmcgcmVsb2FkZWQu
ICBUaGlzIGF2b2lkcyBpbnRlcm1pdC0NCiAgICAgICAgICAgICB0ZW50IGFjY2VzcyBlcnJvcnMg
Zm9yIGNsaWVudHMgdGhhdCBkbyBORlMgUlBDcyB3aGlsZSB0aGUgZXhwb3J0cw0KICAgICAgICAg
ICAgIGFyZSBiZWluZyByZWxvYWRlZCwgYnV0IGludHJvZHVjZXMgYSBkZWxheSBpbiBSUEMgcmVz
cG9uc2Ugd2hpbGUNCiAgICAgICAgICAgICB0aGUgcmVsb2FkIGlzIGluIHByb2dyZXNzLiAgSWYg
bW91bnRkIGNyYXNoZXMgd2hpbGUgYW4gZXhwb3J0cw0KICAgICAgICAgICAgIGxvYWQgaXMgaW4g
cHJvZ3Jlc3MsIG1vdW50ZCBtdXN0IGJlIHJlc3RhcnRlZCB0byBnZXQgdGhlIG5mc2QNCiAgICAg
ICAgICAgICB0aHJlYWRzIHJ1bm5pbmcgYWdhaW4sIGlmIHRoaXMgb3B0aW9uIGlzIHVzZWQuDQoN
Cg0Kc28gZmFyLCBpIHdhcyBhYmxlIHRvIHJlbG9hZCB0aGUgZXhwb3J0cyBsaXN0IHR3aWNlLCB3
aXRob3V0IGFueSByYW5kb20gc3VzcGVuZGVkIHZtLiBsZXRzIHNlZSBpZiB0aGlzIGlzIGEgcmVh
bCBzb2x1dGlvbiBvciBpZiBpIGp1c3QgaGFkIGx1Y2sgdHdvIHRpbWVzLg0KDQoNCkl0IHdvdWxk
IHNlZW0gYXMgaWYgd2UgYXJlIG9uIHRoZSBzYW1lIGJvYXQ6KSBBY3R1YWxseSBJIGhhZG7CtHQg
dGhvdWdodCBhYm91dCBpdCBiZWZvcmUsIGJ1dCB5b3XCtHJlIHJpZ2h0OyBpc3N1aW5nIGEgInNl
cnZpY2UgbW91bnRkIHJlbG9hZCIgZG9lcyBwYXVzZSBhIGxhcmdlIG51bWJlciBvZiBWTcK0cywg
ZnJpY2tpbiBhbm5veWluZyByZWFsbHkuIEkgbWVhbiwgdGhlIE5GUyBzZXJ2ZXIgZG9lc27CtHQg
Y2FyZSB3aGF0IG9yIHdobyBpdMK0cyBzZXJ2aW5nLCB5b3UgY291bGQgYmUgY3JlYXRpbmcgYSBu
ZXcgZXhwb3J0IGZvciBhIGNvbXBsZXRlbHkgZGlmZmVyZW50IHN5c3RlbSwgYW5kIG5vdCBldmVu
IGhhdmUgb1ZpcnQgaW4gbWluZCBiZWZvcmUgY3VzdG9tZXJzIHN0YXJ0IHRvIGNhbGwsIHdvbmRl
cmluZyB3aHkgdGhlaXIgVk3CtHMgaGF2ZSBzdG9wcGVkIHJlc3BvbmRpbmchPw0KDQoNCg0KDQpl
eGFjdGx5IHRoZSBzYW1lIGkgaGF2ZQ0KDQpJIGFjdHVhbGx5IHRyaWVkIHRoYXQgIi1TIiBidXQg
aXQgZGlkbsK0dCB3b3JrIGZvciBtZSBhdCBhbGwsIGFuZCBsb29raW5nIGF0IHRoZSBtYW4tcGFn
ZSBmb3IgbW91bnRkLCB0aGVyZcK0cyBubyBtZW50aW9uIG9mIGl0IGVpdGhlciwgZXZlbiB0aG91
Z2ggd2UgYXJlIHByZXN1bWFibHkgcnVubmluZyB0aGUgc2FtZSB2ZXJzaW9uOg0KIyB1bmFtZSAt
cg0KOS4xLVJFTEVBU0UNCg0KT3IgYXJlIHlvdSBwZXJoYXBzIHRyYWNraW5nICItU1RBQkxFIiwg
YW5kIHRoZXJlwrRzIGEgbWlub3IgZGlmZmVyZW5jZSB0aGVyZT8NCg0KDQoNCg0KDQppIGFtIHRy
YWNraW5nIC1TVEFCTEUsIGJ1dCB0aGUgTWFuIFBhZ2Ugb2YgIm1vdW50ZCIgb24gYSA5LjEgU3Rh
YmxlIChTbmFwc2hvdCBSZWxlYXNlKSBhbHNvIHNob3dzIC1TIFBhcmFtZXRlcg0KDQoNCjkuMS1T
VEFCTEUgRnJlZUJTRCA5LjEtU1RBQkxFICMwOiBTdW4gSnVsICA3IDEwOjUzOjQ2IFVUQyAyMDEz
ICAgICByb290QHNuYXAuZnJlZWJzZC5vcmc6L3Vzci9vYmovdXNyL3NyYy9zeXMvR0VORVJJQyAg
YW1kNjQNCg0KDQoNCjkuMi1QUkVSRUxFQVNFIEZyZWVCU0QgOS4yLVBSRVJFTEVBU0UgIzY6IFRo
dSBKdWwgMTggMDI6NDE6NTcgQ0VTVCAyMDEzICAgICByb290QGZpbGVyMS5pbnRlcm4uPG1haWx0
bzpyb290QGZpbGVyMS5pbnRlcm4uPg0KDQpPSywgc28gd2XCtHJlIG5vdCB1c2luZyB0aGUgc2Ft
ZSB2ZXJzaW9ucywgLVNUQUJMRSAhPSAtUkVMRUFTRSwgYW5kIEkgb25seSB1c2UgLVJFTEVBU0Uu
IEJ1dCB0aGF0IGV4cGxhaW5zIGl0LiBJIGd1ZXNzIEkgY2FuIHdhaXQgZm9yIDkuMi1SRUxFQVNF
IHRvIGdldCByaWQgb2YgdGhhdCBudWlzYW5jZS4gVGhhbmtzIGZvciB0aGUgaW5mbyENCg0KL0th
cmxpDQoNCg0KwrQNCg0KDQpib3RoIHN5c3RlbXMgcHJvdmlkZSAtUyBmb3IgdGhlIG1vdW50ZCwg
YW5kIHNvIGZhciBpIGRpZG50IGhhdmUgYW55IG1vcmUgcHJvYmxlbXMuIGxldHMgc2VlIGlmIHRo
aXMga2VlcHMgZ29pbmcgZ29vZC4NCg0KDQpidXQgaSBhbSBzdGlsbCBpbnRlcmVzdGVkIGluIHBh
cmFtZXRlcnMgd2hpY2ggbWFrZSB0aGUgdmRzbSBtb3JlIHRvbGVyYW50IHRvIHNob3J0IGludGVy
cnVwdGlvbnMuIGluc3RhbnQgc3VzcGVuZCBvZiBhIHZtIGFmdGVyIHN1Y2ggYSBzaG9ydCAib3V0
YWdlIiBpcyBub3QgdmVyeSBuaWNlLg0KDQoNCisxIQ0KDQovS2FybGkNCg0KDQoNCg0KDQpDaGVl
cnMsDQoNCg0KSnVlcmdlbg0KDQoNCi0tDQoNCg0KU2VudCBmcm9tIHRoZSBEZWx0YSBxdWFkcmFu
dCB1c2luZyBCb3JnIHRlY2hub2xvZ3khDQoNCg0KLS0NCg0KTWVkIFbDpG5saWdhIEjDpGxzbmlu
Z2FyDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQpLYXJsaSBTasO2YmVyZw0KU3dlZGlzaCBVbml2
ZXJzaXR5IG9mIEFncmljdWx0dXJhbCBTY2llbmNlcw0KQm94IDcwNzkgKFZpc2l0aW5nIEFkZHJl
c3MgS3JvbsOlc3bDpGdlbiA4KQ0KUy03NTAgMDcgVXBwc2FsYSwgU3dlZGVuDQpQaG9uZTogICs0
Ni0oMCkxOC02NyAxNSA2Ng0Ka2FybGkuc2pvYmVyZ0BzbHUuc2U8bWFpbHRvOmthcmxpLnNqb2Jl
cmdAYWRtLnNsdS5zZT4NCg==
--_000_5F9E965F5A80BC468BE5F40576769F092E6EED87exchange21_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64
PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMCBUUkFOU0lUSU9OQUwv
L0VOIj4NCjxodG1sPg0KPGhlYWQ+DQo8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNv
bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+DQo8bWV0YSBuYW1lPSJHRU5FUkFUT1Ii
IGNvbnRlbnQ9Ikd0a0hUTUwvNC40LjQiPg0KPC9oZWFkPg0KPGJvZHk+DQptw6VuIDIwMTMtMDct
Mjkga2xvY2thbiAxMzoyNiAmIzQzOzAyMDAgc2tyZXYgc3F1YWRyYToNCjxibG9ja3F1b3RlIHR5
cGU9IkNJVEUiPkhpIEthcmxpLCA8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRF
Ij48YnI+DQo8YnI+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj5pIGFs
cmVhZHkgdGhvdWdodCB0aGF0IGkgYW0gdGhlIG9ubHkgb25lIHdpdGggdGhhdCBjb21iaW5hdGlv
biA7KTxicj4NCjwvYmxvY2txdW90ZT4NCjxicj4NCldlbGwsIEkgaGFwcGVuIHRvIGJlIHVzaW5n
IEZlZG9yYSBhcyBlbmdpbmUvaG9zdHMsIGJ1dCB3aGVuIGl0IGNvbWVzIHRvIHRoZSBORlMtc2Vy
dmVyLCB3aHkgc2V0dGxlIGZvciBhbnl0aGluZyBsZXNzLCByaWdodD86KSBJIGltYWdpbmUgeW91
wrRyZSBpbiBpdCBmb3IgdGhlIHNhbWUgcmVhc29uIGFzIG1lIHRvbzsgJnF1b3Q7dGhlIGxhc3Qg
d29yZCBpbiBmaWxlc3lzdGVtcyZxdW90Oy4uLjxicj4NCjxicj4NCjxibG9ja3F1b3RlIHR5cGU9
IkNJVEUiPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUi
Pk9uIE1vbiwgSnVsIDI5LCAyMDEzIGF0IDE6MTEgUE0sIEthcmxpIFNqw7ZiZXJnICZsdDs8YSBo
cmVmPSJtYWlsdG86S2FybGkuU2pvYmVyZ0BzbHUuc2UiPkthcmxpLlNqb2JlcmdAc2x1LnNlPC9h
PiZndDsgd3JvdGU6DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj4NCjxi
bG9ja3F1b3RlPm9ucyAyMDEzLTA3LTI0IGtsb2NrYW4gMjM6MzUgJiM0MzswMjAwIHNrcmV2IHNx
dWFkcmE6IDwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJ
VEUiPg0KPGJsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj5NYXliZSBmb3VuZCBh
IHdvcmthcm91bmQgb24gdGhlIE5GUyBzZXJ2ZXIgc2lkZSwgYSBvcHRpb24gZm9yIHRoZSBtb3Vu
dGQgc2VydmljZTxicj4NCjxicj4NCjxicj4NCjxicj4NCjxicj4NCiZuYnNwOyAmbmJzcDsgJm5i
c3A7LVMgJm5ic3A7ICZuYnNwOyAmbmJzcDtUZWxsIG1vdW50ZCB0byBzdXNwZW5kL3Jlc3VtZSBl
eGVjdXRpb24gb2YgdGhlIG5mc2QgdGhyZWFkcyB3aGVuLTxicj4NCiZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO2V2ZXIgdGhlIGV4cG9ydHMgbGlzdCBpcyBi
ZWluZyByZWxvYWRlZC4gJm5ic3A7VGhpcyBhdm9pZHMgaW50ZXJtaXQtPGJyPg0KJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7dGVudCBhY2Nlc3MgZXJyb3Jz
IGZvciBjbGllbnRzIHRoYXQgZG8gTkZTIFJQQ3Mgd2hpbGUgdGhlIGV4cG9ydHM8YnI+DQombmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDthcmUgYmVpbmcgcmVs
b2FkZWQsIGJ1dCBpbnRyb2R1Y2VzIGEgZGVsYXkgaW4gUlBDIHJlc3BvbnNlIHdoaWxlPGJyPg0K
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7dGhlIHJlbG9h
ZCBpcyBpbiBwcm9ncmVzcy4gJm5ic3A7SWYgbW91bnRkIGNyYXNoZXMgd2hpbGUgYW4gZXhwb3J0
czxicj4NCiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO2xv
YWQgaXMgaW4gcHJvZ3Jlc3MsIG1vdW50ZCBtdXN0IGJlIHJlc3RhcnRlZCB0byBnZXQgdGhlIG5m
c2Q8YnI+DQombmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDt0
aHJlYWRzIHJ1bm5pbmcgYWdhaW4sIGlmIHRoaXMgb3B0aW9uIGlzIHVzZWQuPGJyPg0KPGJyPg0K
PGJyPg0Kc28gZmFyLCBpIHdhcyBhYmxlIHRvIHJlbG9hZCB0aGUgZXhwb3J0cyBsaXN0IHR3aWNl
LCB3aXRob3V0IGFueSByYW5kb20gc3VzcGVuZGVkIHZtLiBsZXRzIHNlZSBpZiB0aGlzIGlzIGEg
cmVhbCBzb2x1dGlvbiBvciBpZiBpIGp1c3QgaGFkIGx1Y2sgdHdvIHRpbWVzLjxicj4NCjwvYmxv
Y2txdW90ZT4NCjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9j
a3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+SXQgd291bGQgc2VlbSBhcyBpZiB3ZSBh
cmUgb24gdGhlIHNhbWUgYm9hdDopIEFjdHVhbGx5IEkgaGFkbsK0dCB0aG91Z2h0IGFib3V0IGl0
IGJlZm9yZSwgYnV0IHlvdcK0cmUgcmlnaHQ7IGlzc3VpbmcgYSAmcXVvdDtzZXJ2aWNlIG1vdW50
ZCByZWxvYWQmcXVvdDsgZG9lcyBwYXVzZSBhIGxhcmdlIG51bWJlciBvZiBWTcK0cywgZnJpY2tp
biBhbm5veWluZyByZWFsbHkuIEkgbWVhbiwgdGhlIE5GUyBzZXJ2ZXIgZG9lc27CtHQgY2FyZSB3
aGF0IG9yDQogd2hvIGl0wrRzIHNlcnZpbmcsIHlvdSBjb3VsZCBiZSBjcmVhdGluZyBhIG5ldyBl
eHBvcnQgZm9yIGEgY29tcGxldGVseSBkaWZmZXJlbnQgc3lzdGVtLCBhbmQgbm90IGV2ZW4gaGF2
ZSBvVmlydCBpbiBtaW5kIGJlZm9yZSBjdXN0b21lcnMgc3RhcnQgdG8gY2FsbCwgd29uZGVyaW5n
IHdoeSB0aGVpciBWTcK0cyBoYXZlIHN0b3BwZWQgcmVzcG9uZGluZyE/PGJyPg0KPGJyPg0KPGJy
Pg0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+
PGJyPg0KPGJyPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+ZXhhY3Rs
eSB0aGUgc2FtZSBpIGhhdmUgPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+
Jm5ic3A7IDwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVv
dGU+SSBhY3R1YWxseSB0cmllZCB0aGF0ICZxdW90Oy1TJnF1b3Q7IGJ1dCBpdCBkaWRuwrR0IHdv
cmsgZm9yIG1lIGF0IGFsbCwgYW5kIGxvb2tpbmcgYXQgdGhlIG1hbi1wYWdlIGZvciBtb3VudGQs
IHRoZXJlwrRzIG5vIG1lbnRpb24gb2YgaXQgZWl0aGVyLCBldmVuIHRob3VnaCB3ZSBhcmUgcHJl
c3VtYWJseSBydW5uaW5nIHRoZSBzYW1lIHZlcnNpb246PGJyPg0KIyB1bmFtZSAtcjxicj4NCjku
MS1SRUxFQVNFPGJyPg0KPGJyPg0KT3IgYXJlIHlvdSBwZXJoYXBzIHRyYWNraW5nICZxdW90Oy1T
VEFCTEUmcXVvdDssIGFuZCB0aGVyZcK0cyBhIG1pbm9yIGRpZmZlcmVuY2UgdGhlcmU/IDwvYmxv
Y2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2Nr
cXVvdGU+PGJyPg0KPGJyPg0KPGJyPg0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJs
b2NrcXVvdGUgdHlwZT0iQ0lURSI+PGJyPg0KPGJyPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVv
dGUgdHlwZT0iQ0lURSI+aSBhbSB0cmFja2luZyAtU1RBQkxFLCBidXQgdGhlIE1hbiBQYWdlIG9m
ICZxdW90O21vdW50ZCZxdW90OyBvbiBhIDkuMSBTdGFibGUgKFNuYXBzaG90IFJlbGVhc2UpIGFs
c28gc2hvd3MgLVMgUGFyYW1ldGVyDQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJD
SVRFIj48YnI+DQo8YnI+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj45
LjEtU1RBQkxFIEZyZWVCU0QgOS4xLVNUQUJMRSAjMDogU3VuIEp1bCAmbmJzcDs3IDEwOjUzOjQ2
IFVUQyAyMDEzICZuYnNwOyAmbmJzcDsgcm9vdEBzbmFwLmZyZWVic2Qub3JnOi91c3Ivb2JqL3Vz
ci9zcmMvc3lzL0dFTkVSSUMgJm5ic3A7YW1kNjQ8YnI+DQo8YnI+DQo8L2Jsb2NrcXVvdGU+DQo8
YmxvY2txdW90ZSB0eXBlPSJDSVRFIj48YnI+DQo8YnI+DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2tx
dW90ZSB0eXBlPSJDSVRFIj45LjItUFJFUkVMRUFTRSBGcmVlQlNEIDkuMi1QUkVSRUxFQVNFICM2
OiBUaHUgSnVsIDE4IDAyOjQxOjU3IENFU1QgMjAxMyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOw0K
PGEgaHJlZj0ibWFpbHRvOnJvb3RAZmlsZXIxLmludGVybi4iPnJvb3RAZmlsZXIxLmludGVybi48
L2E+PGJyPg0KPC9ibG9ja3F1b3RlPg0KPGJyPg0KT0ssIHNvIHdlwrRyZSBub3QgdXNpbmcgdGhl
IHNhbWUgdmVyc2lvbnMsIC1TVEFCTEUgIT0gLVJFTEVBU0UsIGFuZCBJIG9ubHkgdXNlIC1SRUxF
QVNFLiBCdXQgdGhhdCBleHBsYWlucyBpdC4gSSBndWVzcyBJIGNhbiB3YWl0IGZvciA5LjItUkVM
RUFTRSB0byBnZXQgcmlkIG9mIHRoYXQgbnVpc2FuY2UuIFRoYW5rcyBmb3IgdGhlIGluZm8hPGJy
Pg0KPGJyPg0KL0thcmxpPGJyPg0KPGJyPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+PGJyPg0K
PC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+wrQgPC9ibG9ja3F1b3RlPg0K
PGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+PGJyPg0KPGJyPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2Nr
cXVvdGUgdHlwZT0iQ0lURSI+Ym90aCBzeXN0ZW1zIHByb3ZpZGUgLVMgZm9yIHRoZSBtb3VudGQs
IGFuZCBzbyBmYXIgaSBkaWRudCBoYXZlIGFueSBtb3JlIHByb2JsZW1zLiBsZXRzIHNlZSBpZiB0
aGlzIGtlZXBzIGdvaW5nIGdvb2QuJm5ic3A7DQo8L2Jsb2NrcXVvdGU+DQo8YmxvY2txdW90ZSB0
eXBlPSJDSVRFIj4NCjxibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+PGJyPg0K
PGJyPg0KYnV0IGkgYW0gc3RpbGwgaW50ZXJlc3RlZCBpbiBwYXJhbWV0ZXJzIHdoaWNoIG1ha2Ug
dGhlIHZkc20gbW9yZSB0b2xlcmFudCB0byBzaG9ydCBpbnRlcnJ1cHRpb25zLiBpbnN0YW50IHN1
c3BlbmQgb2YgYSB2bSBhZnRlciBzdWNoIGEgc2hvcnQgJnF1b3Q7b3V0YWdlJnF1b3Q7IGlzIG5v
dCB2ZXJ5IG5pY2UuPGJyPg0KPC9ibG9ja3F1b3RlPg0KPGJyPg0KPGJyPg0KPC9ibG9ja3F1b3Rl
Pg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+DQo8YmxvY2txdW90ZT4m
IzQzOzEhJm5ic3A7IDwvYmxvY2txdW90ZT4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5
cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+PGJyPg0KL0thcmxpIDwvYmxvY2txdW90ZT4NCjwvYmxv
Y2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9IkNJVEUiPg0KPGJsb2NrcXVvdGU+PGJyPg0KPGJy
Pg0KPC9ibG9ja3F1b3RlPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+
PGJyPg0KPC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+PGJyPg0KPGJyPg0K
PC9ibG9ja3F1b3RlPg0KPGJsb2NrcXVvdGUgdHlwZT0iQ0lURSI+Q2hlZXJzLCA8L2Jsb2NrcXVv
dGU+DQo8YmxvY2txdW90ZSB0eXBlPSJDSVRFIj48YnI+DQo8YnI+DQo8L2Jsb2NrcXVvdGU+DQo8
YmxvY2txdW90ZSB0eXBlPSJDSVRFIj5KdWVyZ2VuIDwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3Rl
IHR5cGU9IkNJVEUiPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHR5cGU9
IkNJVEUiPi0tDQo8cHJlPgpTZW50IGZyb20gdGhlIERlbHRhIHF1YWRyYW50IHVzaW5nIEJvcmcg
dGVjaG5vbG9neSEKPC9wcmU+DQo8L2Jsb2NrcXVvdGU+DQo8YnI+DQo8dGFibGUgY2VsbHNwYWNp
bmc9IjAiIGNlbGxwYWRkaW5nPSIwIiB3aWR0aD0iMTAwJSI+DQo8dGJvZHk+DQo8dHI+DQo8dGQ+
LS0gPGJyPg0KPGJyPg0KTWVkIFbDpG5saWdhIEjDpGxzbmluZ2FyPGJyPg0KLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLTxicj4NCkthcmxpIFNqw7ZiZXJnPGJyPg0KU3dlZGlzaCBVbml2ZXJzaXR5IG9m
IEFncmljdWx0dXJhbCBTY2llbmNlczxicj4NCkJveCA3MDc5IChWaXNpdGluZyBBZGRyZXNzIEty
b27DpXN2w6RnZW4gOCk8YnI+DQpTLTc1MCAwNyBVcHBzYWxhLCBTd2VkZW48YnI+DQpQaG9uZTog
Jm5ic3A7JiM0Mzs0Ni0oMCkxOC02NyAxNSA2Njxicj4NCjxhIGhyZWY9Im1haWx0bzprYXJsaS5z
am9iZXJnQGFkbS5zbHUuc2UiPmthcmxpLnNqb2JlcmdAc2x1LnNlPC9hPiA8L3RkPg0KPC90cj4N
CjwvdGJvZHk+DQo8L3RhYmxlPg0KPC9ib2R5Pg0KPC9odG1sPg0K
--_000_5F9E965F5A80BC468BE5F40576769F092E6EED87exchange21_--
10:49 p.m.
On Mon, Jul 29, 2013 at 1:46 PM, Karli Sjöberg <Karli.Sjoberg(a)slu.se> wrote:
**
mån 2013-07-29 klockan 13:26 +0200 skrev squadra:
Hi Karli,
i already thought that i am the only one with that combination ;)
Well, I happen to be using Fedora as engine/hosts, but when it comes to
the NFS-server, why settle for anything less, right?:) I imagine you´re in
it for the same reason as me too; "the last word in filesystems"...
exactly :)
On Mon, Jul 29, 2013 at 1:11 PM, Karli Sjöberg <Karli.Sjoberg(a)slu.se>
wrote:
ons 2013-07-24 klockan 23:35 +0200 skrev squadra:
Maybe found a workaround on the NFS server side, a option for the
mountd service
-S Tell mountd to suspend/resume execution of the nfsd threads
when-
ever the exports list is being reloaded. This avoids
intermit-
tent access errors for clients that do NFS RPCs while the
exports
are being reloaded, but introduces a delay in RPC response
while
the reload is in progress. If mountd crashes while an exports
load is in progress, mountd must be restarted to get the nfsd
threads running again, if this option is used.
so far, i was able to reload the exports list twice, without any random
suspended vm. lets see if this is a real solution or if i just had luck two
times.
It would seem as if we are on the same boat:) Actually I hadn´t thought
about it before, but you´re right; issuing a "service mountd reload" does
pause a large number of VM´s, frickin annoying really. I mean, the NFS
server doesn´t care what or who it´s serving, you could be creating a new
export for a completely different system, and not even have oVirt in mind
before customers start to call, wondering why their VM´s have stopped
responding!?
exactly the same i have
I actually tried that "-S" but it didn´t work for me at all, and looking
at the man-page for mountd, there´s no mention of it either, even though we
are presumably running the same version:
# uname -r
9.1-RELEASE
Or are you perhaps tracking "-STABLE", and there´s a minor difference
there?
i am tracking -STABLE, but the Man Page of "mountd" on a 9.1 Stable
(Snapshot Release) also shows -S Parameter
9.1-STABLE FreeBSD 9.1-STABLE #0: Sun Jul 7 10:53:46 UTC 2013
root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
9.2-PRERELEASE FreeBSD 9.2-PRERELEASE #6: Thu Jul 18 02:41:57 CEST
2013 root(a)filer1.intern.
OK, so we´re not using the same versions, -STABLE != -RELEASE, and I only
use -RELEASE. But that explains it. I guess I can wait for 9.2-RELEASE to
get rid of that nuisance. Thanks for the info!
Just checked the changes in -stable, here we go...
http://svnweb.freebsd.org/base/stable/9/usr.sbin/mountd/mountd.c?revision...
9.2 is not so far away :)
Cheers,
Juergen
--
Sent from the Delta quadrant using Borg technology!
3927
days inactive
3932
days old
5 comments
2 participants
participants (2)
-
Karli Sjöberg -
squadra