HostedEngine with HA
by Carlos Rodrigues
Hello,
I have one cluster with two hosts with power management correctly
configured and one virtual machine with HostedEngine over shared
storage with FiberChannel.
When i shutdown the network of host with HostedEngine VM, it should be
possible the HostedEngine VM migrate automatically to another host?
What is the expected behaviour on this HA scenario?
Regards,
--
Carlos Rodrigues
Engenheiro de Software Sénior
Eurotux Informática, S.A. | www.eurotux.com
(t) +351 253 680 300 (m) +351 911 926 110
3 years, 1 month
Very Slow Console Performance - Windows 10
by FERNANDO FREDIANI
This is a multi-part message in MIME format.
--------------C64C2289E18148FC52BD4390
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Hello
Has anyone installed a Windows 10 Virtual Machine ?
I am having serious Console Performance issues even after installing the
Ted Hat QXL controller from the virtio-win ISO.
Someone informed in a forum having similar issues and have resolved by
increasing the graphics card memory to 65536 by editing the XML (example
below), but how is that possible in oVirt permanently ?
<video>
<model type='qxl' ram='131072' vram='131072' vgamem='65536'
heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
Thanks
Fernando
--------------C64C2289E18148FC52BD4390
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font face="arial, helvetica, sans-serif">Hello<br>
<br>
Has anyone installed a Windows 10 Virtual Machine ? <br>
<br>
I am having serious Console Performance issues even after
installing the Ted Hat QXL controller from the virtio-win ISO.<br>
Someone informed in a forum having similar issues and have
resolved by increasing the graphics card memory to 65536 by
editing the XML (example below), but how is that possible in oVirt
permanently ?</font><br>
<br>
<video><br>
<model type='qxl' ram='131072' vram='131072' vgamem='<ins>65536</ins>'
heads='1'/><br>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/><br>
</video><br>
<br>
Thanks<br>
Fernando<br>
</body>
</html>
--------------C64C2289E18148FC52BD4390--
3 years, 2 months
oVIRT 4.1 / iSCSI Multipathing
by Devin Acosta
I am using the latest release of oVIRT 4.1.3, and I am connecting a Dell
Compelent SAN that has 2 fault domains each on a separate VLAN that I have
attached to oVIRT. From what I understand I am suppose to go into “iSCSI
Multipathing” option and add a BOND of the iSCSI interfaces. I have done
this selecting the 2 logical networks together for iSCSI. I notice that
there is an option below to select Storage Targets but if I select the
storage targets below with the logical networks the the cluster goes crazy
and appears to be mad. Storage, Nodes, and everything goes offline even
thought I have NFS also attached to the cluster.
How should this best be configured. What we notice that happens is when the
server reboots it seems to log into the SAN correctly but according the the
Dell SAN it is only logged into once controller. So only pulls both fault
domains from a single controller.
Please Advise.
Devin
3 years, 2 months
when creating VMs, I don't want hosted_storage to be an option
by Mike Farnam
Hi All - Is that a way to mark hosted_storage somehow so that it’s not available to add new VMs to? Right now it’s the default storage domain when adding a VM. At the least, I’d like to make another storage domain the default.
Is there a way to do this?
Thanks
3 years, 2 months
qemu-kvm images corruption
by Nicolas Ecarnot
TL;DR:
How to avoid images corruption?
Hello,
On two of our old 3.6 DC, a recent series of VM migrations lead to some
issues :
- I'm putting a host into maintenance mode
- most of the VM are migrating nicely
- one remaining VM never migrates, and the logs are showing :
* engine.log : "...VM has been paused due to I/O error..."
* vdsm.log : "...Improbable extension request for volume..."
After digging amongst the RH BZ tickets, I saved the day by :
- stopping the VM
- lvchange -ay the adequate /dev/...
- qemu-img check [-r all] /rhev/blahblah
- lvchange -an...
- boot the VM
- enjoy!
Yesterday this worked for a VM where only one error occurred on the qemu
image, and the repair was easily done by qemu-img.
Today, facing the same issue on another VM, it failed because the errors
were very numerous, and also because of this message :
[...]
Rebuilding refcount structure
ERROR writing refblock: No space left on device
qemu-img: Check failed: No space left on device
[...]
The PV/VG/LV are far from being full, so I guess I don't where to look at.
I tried many ways to solve it but I'm not comfortable at all with qemu
images, corruption and solving, so I ended up exporting this VM (to an
NFS export domain), importing it into another DC : this had the side
effect to use qemu-img convert from qcow2 to qcow2, and (maybe?????) to
solve some errors???
I also copied it into another qcow2 file with the same qemu-img convert
way, but it is leading to another clean qcow2 image without errors.
I saw that on 4.x some bugs are fixed about VM migrations, but this is
not the point here.
I checked my SANs, my network layers, my blades, the OS (CentOS 7.2) of
my hosts, but I see nothing special.
The real reason behind my message is not to know how to repair anything,
rather than to understand what could have lead to this situation?
Where to keep a keen eye?
--
Nicolas ECARNOT
3 years, 3 months
Official Hyperconverged Gluster oVirt upgrade procedure?
by Hanson
Hi Guys,
Just wondering if we have an updated manual or whats the current
procedure for upgrading the nodes in a hyperconverged ovirt gluster pool?
Ie Nodes run 4.0 oVirt, as well as GlusterFS, and hosted-engine running
in a gluster storage domain.
Put node in maintenance mode and disable glusterfs from ovirt gui, run
yum update?
Thanks!
3 years, 3 months
[vdsm] status update: running containers alongside VMs
by Francesco Romani
Hi everyone,
I'm happy to share some progress about the former "convirt"[1] project,
which aims to let Vdsm containers alongside VMs, on bare metal.
In the last couple of months I kept updating the patch series, which
is approaching the readiness to be merged in Vdsm.
Please read through this mail to see what the patchset can do now,
how you could try it *now*, even before it is merged.
Everyone is invited to share thoughts and ideas about how this effort
could evolve.
This will be a long mail; I will amend, enhance and polish the content
and make a blog post (on https://mojaves.github.io) to make it easier
to consume and to have some easy-to-find documentation. Later on the
same content will appear also on the oVirt blog.
Happy hacking!
+++
# How to try how the experimental container support for Vdsm.
Vdsm is gaining *experimental* support to run containers alongside VMs.
Vdsm had since long time the ability to manage VMs which run containers,
and recently gained support for
[atomic guests](http://www.projectatomic.io/blog/2015/01/running-ovirt-guest-agent-as-privileged-container/).
With the new support we are describing, you will be able to manage containers
with the same, proven infrastructure that let you manage VMs.
This feature is currently being developed and it is still not merged in the
Vdsm codebase, so some extra work is needed if you want to try it out.
We aiming to merge it in the oVirt 4.1.z cycle.
## What works, aka what to expect
The basic features are expected to work:
1. Run any docker image on the public docker registry
2. Make the container accessible from the outside (aka not just from localhost)
3. Use file-based storage for persistent volumes
## What does not yet work, aka what NOT to expect
Few things are planned and currently under active development:
1. Monitoring. Engine will not get any update from the container besides "VM" status (Up, Down...)
One important drawback is that you will not be told the IP of the container from Engine,
you will need to connect to the Vdsm host to discover it using standard docker tools.
2. Proper network integration. Some steps still need manual intervention
3. Stability and recovery - it's pre-alpha software after all! :)
## 1. Introduction and prerequisites
Trying out container support affects only the host and the Vdsm.
Besides add few custom properties (totally safe and supported since early
3.z), there are zero changes required to the DB and to Engine.
Nevertheless, we recommend to dedicate one oVirt 4.y environment,
or at least one 4.y host, to try out the container feature.
To get started, first thing you need is to setup a vanilla oVirt 4.y
installation. We will need to make changes to the Vdsm and to the
Vdsm host, so hosted engine and/or oVirt node may add extra complexity,
better to avoid them at the moment.
The reminder of this tutorial assumes you are using two hosts,
one for Vdsm (will be changed) and one for Engine (will require zero changes);
furthermore, we assume the Vdsm host is running on CentOS 7.y.
We require:
- one test host for Vdsm. This host need to have one NIC dedicated to containers.
We will use the [docker macvlan driver](https://raesene.github.io/blog/2016/07/23/Docker-MacVLAN/),
so this NIC *must not be* part of one bridge.
- docker >= 1.12
- oVirt >= 4.0.5 (Vdsm >= 4.18.15)
- CentOS >= 7.2
Docker >= 1.12 is avaialable for download [here](https://docs.docker.com/engine/installation/linux/centos/)
Caveats:
1. docker from official rpms conflicts con docker from CentOS, and has a different package name: docker-engine vs docker.
Please note that the kubernetes package from CentOS, for example, require 'docker', not 'docker-engine'.
2. you may want to replace the default service file
[with this one](https://github.com/mojaves/convirt/blob/master/patches/centos72/syst...
and to use this
[sysconfig file](https://github.com/mojaves/convirt/blob/master/patches/centos72/sys....
Here I'm just adding the storage options docker requires, much like the CentOS docker is configured.
Configuring docker like this can save you some troubleshooting, especially if you had docker from CentOS installed
on the testing box.
## 2. Patch Vdsm to support containers
You need to patch and rebuild Vdsm.
Fetch [this patch](https://github.com/mojaves/convirt/blob/master/patches/vdsm/4.18.1...
and apply it against Vdsm 4.18.15.1. Vdsm 4.18.15.{1,2,...} are supported as well.
Rebuild Vdsm and reinstall on your box.
[centos 7.2 packages are here](https://github.com/mojaves/convirt/tree/master/rpms/centos72)
Make sure you install the Vdsm command line client (vdsm-cli)
Restart *both* Vdsm and Supervdsm, make sure Engine still works flawlessly with patched Vdsm.
This ensure that no regression is introduced, and that your environment can run VMs just as before.
Now we can proceed adding the container support.
start docker:
# systemctl start docker-engine
(optional)
# systemctl enable docker-engine
Restart Vdsm again
# systemctl restart vdsm
Now we can check if Vdsm detects docker, so you can use it:
still on the same Vdsm host, run
$ vdsClient -s 0 getVdsCaps | grep containers
containers = ['docker', 'fake']
This means this Vdsm can run containers using 'docker' and 'fake' runtimes.
Ignore the 'fake' runtime; as the name suggests, is a test driver, kinda like /dev/null.
Now we need to make sure the host network configuration is fine.
### 2.1. Configure the docker network for Vdsm
PLEASE NOTE
that the suggested network configuration assumes that
* you have one network, `ovirtmgmt` (the default one) you use for everything
* you have one Vdsm host with at least two NICs, one bound to the `ovirtmgmt` network, and one spare
_This step is not yet automated by Vdsm_, so manual action is needed; Vdsm will take
care of this automatically in the future.
You can use
[this helper script](https://github.com/mojaves/convirt/blob/master/patches/vdsm/cont-...,
which reuses the Vdsm libraries. Make sure
you have patched Vdsm to support container before to use it.
Let's review what the script needs:
# ./cont-setup-net -h
usage: cont-setup-net [-h] [--name [NAME]] [--bridge [BRIDGE]]
[--interface [INTERFACE]] [--gateway [GATEWAY]]
[--subnet [SUBNET]] [--mask [MASK]]
optional arguments:
-h, --help show this help message and exit
--name [NAME] network name to use
--bridge [BRIDGE] bridge to use
--interface [INTERFACE]
interface to use
--gateway [GATEWAY] address of the gateway
--subnet [SUBNET] subnet to use
--mask [MASK] netmask to use
So we need to feed --name, --interface, --gateway, --subnet and optionally --mask (default, /24, is often fine).
For my case the default mask was indeed fine, so I used the script like this:
# ./cont-setup-net --name ovirtmgmt --interface enp3s0 --gateway 192.168.1.1 --subnet 192.168.1.0
Thhis is the output I got:
DEBUG:virt.containers.runtime:configuring runtime 'docker'
DEBUG:virt.containers.command:* calling ['/bin/docker', 'network', 'inspect', 'ovirtmgmt']
Error: No such network: ovirtmgmt
DEBUG:virt.containers.command:* called ['/bin/docker', 'network', 'inspect', 'ovirtmgmt']
DEBUG:virt.containers.runtime.Docker:config: cannot load 'ovirtmgmt', ignored
DEBUG:virt.containers.command:* calling ['/bin/docker', 'network', 'create', '-d', 'macvlan', '--subnet=192.168.1.0/24', '--gateway=192.168.1.1', '--ip-range=192.168.1.0/24', '-o', 'parent=enp3s0', 'ovirtmgmt']
DEBUG:virt.containers.command:* called ['/bin/docker', 'network', 'create', '-d', 'macvlan', '--subnet=192.168.1.0/24', '--gateway=192.168.1.1', '--ip-range=192.168.1.0/24', '-o', 'parent=enp3s0', 'ovirtmgmt']
DEBUG:virt.containers.runtime:configuring runtime 'fake'
You can clearly see what the script did, and why it needed the root privileges. Let's deoublecheck using the docker tools:
# docker network ls
NETWORK ID NAME DRIVER SCOPE
91535f3425a8 bridge bridge local
d42f7e5561b5 host host local
621ab6dd49b1 none null local
f4b88e4a67eb ovirtmgmt macvlan local
# docker network inspect ovirtmgmt
[
{
"Name": "ovirtmgmt",
"Id": "f4b88e4a67ebb7886ec74073333d613b1893272530cae4d407c95ab587c5fea1",
"Scope": "local",
"Driver": "macvlan",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "192.168.1.0/24",
"IPRange": "192.168.1.0/24",
"Gateway": "192.168.1.1"
}
]
},
"Internal": false,
"Containers": {},
"Options": {
"parent": "enp3s0"
},
"Labels": {}
}
]
Looks good! the host configuration is completed. Let's move to the Engine side.
## 3. Configure Engine
As mentioned above, we need now to configure Engine. This boils down to:
Add a few custom properties for VMs:
In case you were already using custom properties, you need to amend the command
line to not overwrite your existing ones.
# engine-config -s UserDefinedVMProperties='volumeMap=^[a-zA-Z_-]+:[a-zA-Z_-]+$;containerImage=^[a-zA-Z]+(://|)[a-zA-Z]+$;containerType=^(docker|rkt)$' --cver=4.0
It is worth stressing that while the variables are container-specific,
the VM custom properties are totally inuntrusive and old concept in oVirt, so
this step is totally safe.
Now restart Engine to let it use the new variables:
# systemctl restart ovirt-engine
The next step is actually configure one "container VM" and run it.
## 4. Create the container "VM"
To finally run a container, you start creating a VM much like you always did, with
few changes
1. most of the hardware-related configuration isn't relevant for container "VMs",
besides cpu share and memory limits; this will be better documented in the
future; unneeded configuration will just be ignored
2. You need to set some custom properties for your container "VM". Those are
actually needed to enable the container flow, and they are documented in
the next section. You *need* to set at least `containerType` and `containerImage`.
### 4.2. Custom variables for container support
The container support needs some custom properties to be properly configured:
1. `containerImage` (*needed* to enable the container system).
Just select the target image you want to run. You can use the standard syntax of the
container runtimes.
2. `containerType` (*needed* to enable the container system).
Selects the container runtime you want to use. All the available options are always showed.
Please note that unavailable container options are not yet grayed out.
If you *do not* have rkt support on your host, you still can select it, but it won't work.
3. `volumeMap` key:value like. You can map one "VM" disk (key) to one container volume (value),
to have persistent storage. Only file-based storage is supported.
Example configuration:
`containerImage = redis`
`containerType = docker`
`volumeMap = vda:data` (this may not be needed, and the volume label is just for illustrative purposes)
### 4.2. A little bit of extra work: preload the images on the Vdsm host
This step is not needed by the flow, and will be handled by oVirt in the future.
The issue is how the container image are handled. They are stored by the container
management system (rkt, docker) on each host, and they are not pre-downloaded.
To shorten the duration of the first boot, you are advised to pre-download
the image(s) you want to run. For example
## on the Vdsm host you want to use with containers
# docker pull redis
## 5. Run the container "VM"
You are now all set to run your "VM" using oVirt Engine, just like any existing VM.
Some actions doesn't make sense for a container "VM", like live migration.
Engine won't stop you to try to do those actions, but they will fail gracefully
using the standard errors.
## 6. Next steps
What to expect from this project in the future?
For the integration with Vdsm, we want to fix the existing known issues, most notably:
* add proper monitoring/reporting of the container health
* ensure proper integration of the container image store with oVirt storage management
* streamline the network configuration
What is explicitely excluded yet is any Engine change. This is a Vdsm-only change at the
moment, so fixing the following is currently unplanned:
* First and foremost, Engine will not distinguish between real VMs and container VMs.
Actions unavailable to container will not be hidden from UI. Same for monitoring
and configuration data, which will be ignored.
* Engine is NOT aware of the volumes one container can use. You must inspect and do the
mapping manually.
* Engine is NOT aware of the available container runtimes. You must select it carefully
Proper integration with Engine may be added in the future once this feature exits
from the experimental/provisional stage.
Thanks for reading, make sure to share your thoughts on the oVirt mailing lists!
+++
[1] we keep calling it that way _only_ internally, because it's a short
name we are used to. After the merge/once we release it, we will use
a different name, like "vdsm-containers" or something like it.
--
Francesco Romani
Red Hat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
3 years, 4 months
Empty cgroup files on centos 7.3 host
by Florian Schmid
Hi,
I wanted to monitor disk IO and R/W on all of our oVirt centos 7.3 hypervisor hosts, but it looks like that all those files are empty.
For example:
ls -al /sys/fs/cgroup/blkio/machine.slice/machine-qemu\\x2d14\\x2dHostedEngine.scope/
insgesamt 0
drwxr-xr-x. 2 root root 0 30. Mai 10:09 .
drwxr-xr-x. 16 root root 0 26. Jun 09:25 ..
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_merged
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_merged_recursive
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_queued
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_queued_recursive
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_service_bytes
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_service_bytes_recursive
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_serviced
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_serviced_recursive
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_service_time
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_service_time_recursive
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_wait_time
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.io_wait_time_recursive
-rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.leaf_weight
-rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.leaf_weight_device
--w-------. 1 root root 0 30. Mai 10:09 blkio.reset_stats
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.sectors
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.sectors_recursive
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.io_service_bytes
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.io_serviced
-rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.read_bps_device
-rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.read_iops_device
-rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.write_bps_device
-rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.throttle.write_iops_device
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.time
-r--r--r--. 1 root root 0 30. Mai 10:09 blkio.time_recursive
-rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.weight
-rw-r--r--. 1 root root 0 30. Mai 10:09 blkio.weight_device
-rw-r--r--. 1 root root 0 30. Mai 10:09 cgroup.clone_children
--w--w--w-. 1 root root 0 30. Mai 10:09 cgroup.event_control
-rw-r--r--. 1 root root 0 30. Mai 10:09 cgroup.procs
-rw-r--r--. 1 root root 0 30. Mai 10:09 notify_on_release
-rw-r--r--. 1 root root 0 30. Mai 10:09 tasks
I thought, I can get my needed values from there, but all files are empty.
Looking at this post: http://lists.ovirt.org/pipermail/users/2017-January/079011.html
this should work.
Is this normal on centos 7.3 with oVirt installed? How can I get those values, without monitoring all VMs directly?
oVirt Version we use:
4.1.1.8-1.el7.centos
BR Florian
3 years, 4 months
Failed to open grubx64.efi
by Julio Cesar Bustamante
Hi there,
I have installed Ovirt Host in a HS22 Blade Ibm, but I have this bug.
Failed to open \efi\centos\grubx64.efi not found
Falied to load image \EFI\centos\grubx64.efi Not found
You know how can I solve this issue ?
--
Julio Cesar Bustamante.
3 years, 4 months
slow performance with export storage on glusterfs
by Jiří Sléžka
This is a cryptographically signed message in MIME format.
--------------ms020509020901010502070100
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Hi,
I am trying realize why is exporting of vm to export storage on
glusterfs such slow.
I am using oVirt and RHV, both instalations on version 4.1.7.
Hosts have dedicated nics for rhevm network - 1gbps, data storage itself
is on FC.
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks
but I can achieve about 200-400mbit throughput in other applications (we
are using it for "cold" data, backups mostly).
I am using this glusterfs cluster as backend for export storage. When I
am exporting vm I can see only about 60-80mbit throughput.
What could be the bottleneck here?
Could it be qemu-img utility?
vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06
/usr/bin/qemu-img convert -p -t none -T none -f raw
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426=
-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d=
6cf-48cc-ad0c-4a7d979c0c1e
-O raw
/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4e=
a2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f=
-d6cf-48cc-ad0c-4a7d979c0c1e
Any idea how to make it work faster or what throughput should I expected?=
Cheers,
Jiri
--------------ms020509020901010502070100
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ
BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy
dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx
MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I
b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd
VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty
qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+
0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim
atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg
oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj
ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB
AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw
AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB
LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD
ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t
L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr
BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj
egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI
hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF
OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS
WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB
iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101
S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow
ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV
BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN
BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw
HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y
ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE
BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT
E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/
VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN
MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E
8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX
aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh
tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf
BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V
jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow
DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE
LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f
BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy
c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj
aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw
Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj
ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC
AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf
/SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx
VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts
UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd
M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB
MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB
bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl
cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG
SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyMDE1MjAzM1owLwYJ
KoZIhvcNAQkEMSIEILmfZ4a/vSZv57MxXgCAPwjxh9wMgWdW8v78HTqqQ3zIMGwGCSqGSIb3
DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG
9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG
AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES
MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl
U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC
CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE
BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j
ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBAGii
FiUp6Lb3R5JRu32BeDHrAGBL8Ym2pCdXh8zv4dYs3Q3dd/SQHkif4Z3qysUHfPAEQ2gz7N4C
W7ALZjblD0RZRy5JlrzGLfCmruPBKmU/cLIAYZk2v78+aEjgi8oiZHSLnhIwCRQheQtxK7rG
O8qkWpqZ2jmCstaBlj8kbpVb4WvNC7EbtEJLjXTP4rKAET2daeZdLqPaLx6eYND3l10GymKV
k4KLh0o638Eh3fmBLIncOh79nuURZl5Q6K7bflEQacQEzWDw9+9fGf5vWc8FzTzrn6w9RPwZ
NCcAVdR8SAU274u0cFWPnj+UywsJRKywJ31UWbQGU8ONy7MsAoAAAAAAAAA=
--------------ms020509020901010502070100--
3 years, 5 months
