- Users - oVirt List Archives

Strange network performance on VirtIIO VM NIC
by FERNANDO FREDIANI 21 Mar '17

21 Mar '17

Hello all. I have a peculiar problem here which perhaps others may have had or know about and can advise. I have Virtual Machine with 2 VirtIO NICs. This VM serves around 1Gbps of traffic with thousands of clients connecting to it. When I do a packet loss test to the IP pinned to NIC1 it varies from 3% to 10% of packet loss. When I run the same test on NIC2 the packet loss is consistently 0%. >From what I gather I may have something to do with possible lack of Multi Queu VirtIO where NIC1 is managed by a single CPU which might be hitting 100% and causing this packet loss. Looking at this reference ( https://fedoraproject.org/wiki/Features/MQ_virtio_net) I see one way to test it is start the VM with 4 queues (for example), but checking on the qemu-kvm process I don't see option present. Any way I can force it from the Engine ? This other reference ( https://www.linux-kvm.org/page/Multiqueue#Enable_MQ_feature) points to the same direction about starting the VM with queues=N Also trying to increase the TX ring buffer within the guest with ethtool -g eth0 is not possible. Oh, by the way, the Load on the VM is significantly high despite the CPU usage isn't above 50% - 60% in average. Thanks Fernando

3 9

How do you oVirt?
by Sandro Bonazzola 21 Mar '17

21 Mar '17

As we continue to develop oVirt 4.2 and future releases, the Development and Integration teams at Red Hat would value insights on how you are deploying the oVirt environment. Please help us to hit the mark by completing this short survey. Survey will close on April 15th Here's the link to the survey: https://docs.google.com/forms/d/e/1FAIpQLSdloxiIP2HrW2HguU0UVbNtKgpSBaJXj-Z… Thanks, -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

Changing gateway ping address
by Matteo 21 Mar '17

21 Mar '17

Hi all, I need to change the gateway ping address, the one used by hosted engine setup. Is ok to edit /etc/ovirt-hosted-engine/hosted-engine.conf on each node, update the gateway param with the new ip address and restart the agent&broker on each node? With a blind test seems ok, but need to understand if is the right procedure. Thanks, Matteo

4 7

error on live migraion: ovn?
by Gianluca Cecchi 21 Mar '17

21 Mar '17

Hello, environment on 4.1. I have a Vm with 2 nics: one of them is on ovn. Trying to migrate I get error. Ho wto decode it? Is live migration supported with OVN or mixed nics? Thanks, Gianluca in engine.log 2017-03-21 10:37:26,209+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-26) [f0cea507-608b-4f36-ac4e-606faec35cd9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[2e571c77-bae1-4c1c-bf98-effaf9fed741=<VM, ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED$VmName c7service>]', sharedLocks='null'}' 2017-03-21 10:37:26,499+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (org.ovirt.thread.pool-6-thread-43) [f0cea507-608b-4f36-ac4e-606faec35cd9] Running command: MigrateVmToServerCommand internal: false. Entities affected : ID: 2e571c77-bae1-4c1c-bf98-effaf9fed741 Type: VMAction group MIGRATE_VM with role type USER 2017-03-21 10:37:26,786+01 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-43) [f0cea507-608b-4f36-ac4e-606faec35cd9] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='30677d2c-4eb8-4ed9-ba54-0b89945a45fd', vmId='2e571c77-bae1-4c1c-bf98-effaf9fed741', srcHost='ovmsrv07.mydomain', dstVdsId='02bb501a-b641-4ee1-bab1-5e640804e65f', dstHost='ovmsrv05.mydomain:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 14390155 2017-03-21 10:37:26,787+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-43) [f0cea507-608b-4f36-ac4e-606faec35cd9] START, MigrateBrokerVDSCommand(HostName = ovmsrv07, MigrateVDSCommandParameters:{runAsync='true', hostId='30677d2c-4eb8-4ed9-ba54-0b89945a45fd', vmId='2e571c77-bae1-4c1c-bf98-effaf9fed741', srcHost='ovmsrv07.mydomain', dstVdsId='02bb501a-b641-4ee1-bab1-5e640804e65f', dstHost='ovmsrv05.mydomain:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 5fd9f196 2017-03-21 10:37:27,386+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-43) [f0cea507-608b-4f36-ac4e-606faec35cd9] FINISH, MigrateBrokerVDSCommand, log id: 5fd9f196 2017-03-21 10:37:27,445+01 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-43) [f0cea507-608b-4f36-ac4e-606faec35cd9] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 14390155 2017-03-21 10:37:27,475+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [f0cea507-608b-4f36-ac4e-606faec35cd9] EVENT_ID: VM_MIGRATION_START(62), Correlation ID: f0cea507-608b-4f36-ac4e-606faec35cd9, Job ID: 514c0562-de81-44a1-bde3-58027662b536, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: c7service, Source: ovmsrv07, Destination: ovmsrv05, User: g.cecchi@internal-authz). 2017-03-21 10:37:30,341+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler4) [53ea57f2] START, FullListVDSCommand(HostName = ovmsrv07, FullListVDSCommandParameters:{runAsync='true', hostId='30677d2c-4eb8-4ed9-ba54-0b89945a45fd', vmIds='[2e571c77-bae1-4c1c-bf98-effaf9fed741]'}), log id: 46e85505 2017-03-21 10:37:30,526+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler4) [53ea57f2] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=2e571c77-bae1-4c1c-bf98-effaf9fed741, guestDiskMapping={0QEMU_QEMU_HARDDISK_6af3dfe5-6da7-48e3-9={name=/dev/sda}, QEMU_DVD-ROM_QM00003={name=/dev/sr0}}, transparentHugePages=true, timeOffset=0, cpuType=Opteron_G2, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@31e4f47d, smartcardEnable=false, custom={device_39acb1f5-c31e-4810-a4c3-26d460a6e374=VmDevice:{id='VmDeviceId:{deviceId='39acb1f5-c31e-4810-a4c3-26d460a6e374', vmId='2e571c77-bae1-4c1c-bf98-effaf9fed741'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_39acb1f5-c31e-4810-a4c3-26d460a6e374device_c45a9213-8f12-46c7-b749-27a4fcb02c67=VmDevice:{id='VmDeviceId:{deviceId='c45a9213-8f12-46c7-b749-27a4fcb02c67', vmId='2e571c77-bae1-4c1c-bf98-effaf9fed741'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_39acb1f5-c31e-4810-a4c3-26d460a6e374device_c45a9213-8f12-46c7-b749-27a4fcb02c67device_22513430-5212-4d6a-a413-4c42a216dc9ddevice_f22e3921-5426-403e-9dab-eaf23b0c628e=VmDevice:{id='VmDeviceId:{deviceId='f22e3921-5426-403e-9dab-eaf23b0c628e', vmId='2e571c77-bae1-4c1c-bf98-effaf9fed741'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_39acb1f5-c31e-4810-a4c3-26d460a6e374device_c45a9213-8f12-46c7-b749-27a4fcb02c67device_22513430-5212-4d6a-a413-4c42a216dc9d=VmDevice:{id='VmDeviceId:{deviceId='22513430-5212-4d6a-a413-4c42a216dc9d', vmId='2e571c77-bae1-4c1c-bf98-effaf9fed741'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=c7service, nice=0, status=Migration Source, maxMemSize=16384, bootMenuEnable=false, pid=4504, smpThreadsPerCore=1, memGuaranteedSize=2048, kvmEnable=true, pitReinjection=false, displayNetwork=ovirtmgmt, devices=[Ljava.lang.Object;@3f3610df, display=qxl, maxVCpus=16, clientIp=, statusTime=4533149590, maxMemSlots=16}], log id: 46e85505 2017-03-21 10:37:30,536+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [53ea57f2] Received a console Device without an address when processing VM 2e571c77-bae1-4c1c-bf98-effaf9fed741 devices, skipping device: {device=console, specParams={consoleType=serial, enableSocket=true}, type=console, deviceId=dcfbd20d-65f4-47d3-ac7f-5c2cc60e561f, alias=serial0} 2017-03-21 10:37:30,536+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [53ea57f2] Received a spice Device without an address when processing VM 2e571c77-bae1-4c1c-bf98-effaf9fed741 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.4.168.76, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=5cf3ed21-1471-44d0-81d8-653044fdf51f, tlsPort=5900} 2017-03-21 10:37:30,537+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [53ea57f2] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='dcfbd20d-65f4-47d3-ac7f-5c2cc60e561f', vmId='2e571c77-bae1-4c1c-bf98-effaf9fed741'}', device='console', type='CONSOLE', bootOrder='0', specParams='[enableSocket=true, consoleType=serial]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}' 2017-03-21 10:37:40,966+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler10) [1949f0c7] Fetched 2 VMs from VDS '02bb501a-b641-4ee1-bab1-5e640804e65f' 2017-03-21 10:37:40,968+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [1949f0c7] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'(c7service) was unexpectedly detected as 'MigratingTo' on VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) (expected on '30677d2c-4eb8-4ed9-ba54-0b89945a45fd') 2017-03-21 10:37:40,968+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [1949f0c7] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741' is migrating to VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) ignoring it in the refresh until migration is done 2017-03-21 10:37:55,992+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [560703e7] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'(c7service) was unexpectedly detected as 'MigratingTo' on VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) (expected on '30677d2c-4eb8-4ed9-ba54-0b89945a45fd') 2017-03-21 10:37:55,992+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [560703e7] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741' is migrating to VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) ignoring it in the refresh until migration is done 2017-03-21 10:38:11,018+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [560703e7] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'(c7service) was unexpectedly detected as 'MigratingTo' on VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) (expected on '30677d2c-4eb8-4ed9-ba54-0b89945a45fd') 2017-03-21 10:38:11,018+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [560703e7] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741' is migrating to VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) ignoring it in the refresh until migration is done 2017-03-21 10:38:26,345+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [707b57d] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'(c7service) was unexpectedly detected as 'MigratingTo' on VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) (expected on '30677d2c-4eb8-4ed9-ba54-0b89945a45fd') 2017-03-21 10:38:26,345+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [707b57d] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741' is migrating to VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) ignoring it in the refresh until migration is done 2017-03-21 10:38:42,371+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [44770476] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'(c7service) was unexpectedly detected as 'MigratingTo' on VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) (expected on '30677d2c-4eb8-4ed9-ba54-0b89945a45fd') 2017-03-21 10:38:42,371+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [44770476] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741' is migrating to VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) ignoring it in the refresh until migration is done 2017-03-21 10:38:57,397+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [46677852] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'(c7service) was unexpectedly detected as 'MigratingTo' on VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) (expected on '30677d2c-4eb8-4ed9-ba54-0b89945a45fd') 2017-03-21 10:38:57,397+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [46677852] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741' is migrating to VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) ignoring it in the refresh until migration is done 2017-03-21 10:39:12,632+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [707b57d] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'(c7service) was unexpectedly detected as 'MigratingTo' on VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) (expected on '30677d2c-4eb8-4ed9-ba54-0b89945a45fd') 2017-03-21 10:39:12,632+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [707b57d] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741' is migrating to VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) ignoring it in the refresh until migration is done 2017-03-21 10:39:12,849+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-1) [] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741' was reported as Down on VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) 2017-03-21 10:39:12,849+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-1) [] START, DestroyVDSCommand(HostName = ovmsrv05, DestroyVmVDSCommandParameters:{runAsync='true', hostId='02bb501a-b641-4ee1-bab1-5e640804e65f', vmId='2e571c77-bae1-4c1c-bf98-effaf9fed741', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 6fa6dc7f 2017-03-21 10:39:13,856+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-1) [] Failed to destroy VM '2e571c77-bae1-4c1c-bf98-effaf9fed741' because VM does not exist, ignoring 2017-03-21 10:39:13,856+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-1) [] FINISH, DestroyVDSCommand, log id: 6fa6dc7f 2017-03-21 10:39:13,856+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-1) [] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'(c7service) was unexpectedly detected as 'Down' on VDS '02bb501a-b641-4ee1-bab1-5e640804e65f'(ovmsrv05) (expected on '30677d2c-4eb8-4ed9-ba54-0b89945a45fd') 2017-03-21 10:39:22,138+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [707b57d] VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'(c7service) moved from 'MigratingFrom' --> 'Up' 2017-03-21 10:39:22,139+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [707b57d] Adding VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'(c7service) to re-run list 2017-03-21 10:39:22,161+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (DefaultQuartzScheduler7) [707b57d] Rerun VM '2e571c77-bae1-4c1c-bf98-effaf9fed741'. Called from VDS 'ovmsrv07' 2017-03-21 10:39:22,273+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [707b57d] START, MigrateStatusVDSCommand(HostName = ovmsrv07, MigrateStatusVDSCommandParameters:{runAsync='true', hostId='30677d2c-4eb8-4ed9-ba54-0b89945a45fd', vmId='2e571c77-bae1-4c1c-bf98-effaf9fed741'}), log id: 53f46b57 2017-03-21 10:39:22,577+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [707b57d] FINISH, MigrateStatusVDSCommand, log id: 53f46b57 2017-03-21 10:39:22,589+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [707b57d] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Correlation ID: f0cea507-608b-4f36-ac4e-606faec35cd9, Job ID: 514c0562-de81-44a1-bde3-58027662b536, Call Stack: null, Custom Event ID: -1, Message: Migration failed (VM: c7service, Source: ovmsrv07, Destination: ovmsrv05). 2017-03-21 10:39:22,595+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (org.ovirt.thread.pool-6-thread-36) [707b57d] Lock freed to object 'EngineLock:{exclusiveLocks='[2e571c77-bae1-4c1c-bf98-effaf9fed741=<VM, ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED$VmName c7service>]', sharedLocks='null'}' 2017-03-21 10:39:27,658+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler2) [560703e7] Fetched 1 VMs from VDS '02bb501a-b641-4ee1-bab1-5e640804e65f' [root@ovmgr1 ovirt-engine-notifier]# on target host vdsm.log some of these (perhaps related to retries): 017-03-21 10:37:39,826 INFO (jsonrpc/4) [dispatcher] Run and protect: repoStats, Return response: {u'5e11aa05-9a40-43d8-b173 -c2294cb0c1be': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00054575', 'lastCheck': '5.0', 'valid' : True}, u'8846db3f-6e07-4468-bbfc-bf5fbc3f8eef': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0006 64548', 'lastCheck': '5.0', 'valid': True}, u'5ed04196-87f1-480e-9fee-9dd450a3b53b': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000413433', 'lastCheck': '7.0', 'valid': True}, u'922b5269-ab56-4c4d-838f-49d33427e2ab': {'code ': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00136826', 'lastCheck': '3.3', 'valid': True}} (logUtils:52) 2017-03-21 10:37:39,839 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.02 seconds (__init__:5 15) 2017-03-21 10:37:40,953 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:552) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 547, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 202, in _dynamicMethod result = fn(*methodArgs) File "/usr/share/vdsm/API.py", line 1410, in getAllVmIoTunePolicies io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() File "/usr/share/vdsm/clientIF.py", line 447, in getAllVmIoTunePolicies vm_io_tune_policies[v.id] = {'policy': v.getIoTunePolicy(), File "/usr/share/vdsm/virt/vm.py", line 2730, in getIoTunePolicy qos = self._getVmPolicy() File "/usr/share/vdsm/virt/vm.py", line 2704, in _getVmPolicy metadata_xml = self._dom.metadata( File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__ % self.vmid) NotConnectedError: VM u'2e571c77-bae1-4c1c-bf98-effaf9fed741' was not started yet or was shut down 2017-03-21 10:37:40,954 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies failed (error -32603) in 0.00 seconds (__init__:515) 2017-03-21 10:37:40,959 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:515) 2017-03-21 10:38:10,999 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:552) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 547, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 202, in _dynamicMethod result = fn(*methodArgs) File "/usr/share/vdsm/API.py", line 1410, in getAllVmIoTunePolicies io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() File "/usr/share/vdsm/clientIF.py", line 447, in getAllVmIoTunePolicies vm_io_tune_policies[v.id] = {'policy': v.getIoTunePolicy(), File "/usr/share/vdsm/virt/vm.py", line 2730, in getIoTunePolicy qos = self._getVmPolicy() File "/usr/share/vdsm/virt/vm.py", line 2704, in _getVmPolicy metadata_xml = self._dom.metadata( File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__ % self.vmid) NotConnectedError: VM u'2e571c77-bae1-4c1c-bf98-effaf9fed741' was not started yet or was shut down

1 0

Event History for a VM
by Sven Achtelik 21 Mar '17

21 Mar '17

--_000_BFAB40933B3367488CE6299BAF8592D1014E52AE6F20SOCRATESasl_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi All, I would need to have an Event-History of our VMs for auditing purposes that= is able to go back until the moment the VM was created/imported. I found t= he Events Tab in the VM view and found that this is not showing everything = to the moment of creation. Things that are important for me would be any ch= ange in CPUs or Host that the VM is pinned to. Are the Events stored in the= Engine DB and can I read them in any way ? Is there a value that needs to = be changed in order to keep all Events for a VM ? Thank you for helping, Sven --_000_BFAB40933B3367488CE6299BAF8592D1014E52AE6F20SOCRATESasl_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV=3D"Content-Type" CONTENT= =3D"text/html; charset=3Dus-ascii"><meta name=3DGenerator content=3D"Micros= oft Word 15 (filtered medium)"><style></style></head><body lang=3DDE link=3D"#0563C1" v= link=3D"#954F72"><div class=3DWordSection1>Hi All, <o:p></o:p><o:p> </o:p>I w= ould need to have an Event-History of our VMs for auditing purposes that is= able to go back until the moment the VM was created/imported. I found the = Events Tab in the VM view and found that this is not showing everything to = the moment of creation. Things that are important for me would be any chang= e in CPUs or Host that the VM is pinned to. Are the Events stored in the En= gine DB and can I read them in any way ? Is there a value that needs to be = changed in order to keep all Events for a VM ? <o:p></o:p><o:p> </o:p>Thank you for helping, <o:p></o:p><o:p> </o:p>Sven<o:p></o:p><o:p> </o:p></div></body></html>= --_000_BFAB40933B3367488CE6299BAF8592D1014E52AE6F20SOCRATESasl_--

4 5

Get Involved with the oVirt Project! Spring 2017 Edition
by Sandro Bonazzola 21 Mar '17

21 Mar '17

Hi, Got some time to spare? Join us on the oVirt project! oVirt is a feature rich server virtualization management system with advanced capabilities. It also provides a crucial venue for user and developer cooperation, and is the first truly open and comprehensive data center virtualization management initiative. If you are new to oVirt, we recommend that you visit the develop <https://www.ovirt.org/develop/> page to learn more about the project, and the working with Gerrit <https://www.ovirt.org/develop/dev-process/working-with-gerrit/> page for more about our code review system. If you haven’t worked with Gerrit before, we recommend you visit the introduction to Gerrit <https://gerrit-review.googlesource.com/Documentation> page. On our community page <https://www.ovirt.org/community/> you’ll discover the many ways in which you can contribute to the project, even without programming skills. More suggestions? Below you’ll find other ways in which you can contribute to the oVirt project. Keep ‘em Fresh Like the idea of having fresh disk images of your favorite distribution in the oVirt Glance repository? You can help in two ways: First, test existing online images to ensure that they work with cloud-init. Second, create an online image and report your success to devel(a)ovirt.org. We’ll be happy to upload the images once they are ready. Get Vdsm Running on Debian If you like Debian and have programming or packaging skills, you can help with the ongoing effort to get VDSM running on Debian <https://www.ovirt.org/develop/release-management/features/debian-support-fo…>. All the current work on Debian (VDSM and related packages) can be found in the Debian git repositories <http://git.debian.org/>. You can also keep up to date on how the work is progressing by subscribing to the oVirt devel mailing list <http://lists.ovirt.org/mailman/listinfo/devel>. Fix Some Bugs These bugs are just waiting to be resolved: Bug ID Summary 1411133 Hosted-Engine: iscsi activation storage failing 1209881 [RFE] Remove iptables from hosted-engine.spec file to be able to deploy hosted-engine without firewall services installed 1353713 [RFE] - iSCSI Setup Should use different User/Password For Discovery and Portal 1130445 [TEXT] - If engine-setup asks for updates and choice is no, suggest '--offline' on re-run. 1174236 [RFE] Integrate installation with Server Roles of Fedora Server 1356425 [TEXT] 'hosted-engine --vm-start' said it destroyed the VM 1328488 The engine fails with oracle java8 Port oVirt to Your Favorite Distribution Are you sure-handed at packaging software, and using a distribution currently unsupported by oVirt? Help to get oVirt ported to any of the following distributions: Fedora <http://lists.ovirt.org/pipermail/devel/2015-September/011426.html> CentOS <https://wiki.centos.org/SpecialInterestGroup/Virtualization> Gentoo <https://wiki.gentoo.org/wiki/OVirt>. Also, check out Google Summer of Code <https://wiki.gentoo.org/wiki/Google_Summer_of_Code/2016/Ideas> to learn how you can contribute to Gentoo and get paid for it! Debian <http://www.ovirt.org/develop/release-management/features/debian-support-for…>: Archlinux <http://www.ovirt.org/develop/developer-guide/arch-linux/> OpenSUSE <https://build.opensuse.org/project/show/Virtualization:oVirt> Get into DevOps If you love DevOps and find yourself counting stable builds in Jenkins CI while trying to fall asleep, then the oVirt infrastructure team is looking for you! Join us and dive into the latest and coolest DevOps tools today! Check out these open tasks <https://ovirt-jira.atlassian.net/secure/RapidBoard.jspa?rapidView=6.>. You can also help by telling us how you use oVirt in your DevOps environment. Email us at devel(a)ovirt.org (Please use [DevOps] in the subject line. For more information on oVirt DevOps, visit oVirt infrastructure docs <http://ovirt-infra-docs.readthedocs.io/en/latest/> and oVirt infra documentation <http://www.ovirt.org/develop/infra/infrastructure-documentation/>. Improve the oVirt System Testing Make oVirt system testing <https://gerrit.ovirt.org/gitweb?p=ovirt-system-tests.git;a=summary> - and the Lago system <https://github.com/lago-project/lago> that runs it - even better. Visit Lago project documentation <http://lago.readthedocs.io/en/stable/> to learn how to run the test suite yourself. More Bugs, No Coding Required No time for DevOps and you are not a programer? You can still contribute. Here are some easy bugs to fix <https://bugzilla.redhat.com/buglist.cgi?quicksearch=classification%3Aovirt%…> that don’t require a single line of code. Tests to Run Do you prefer to test things? Here are some test cases you can try <http://www.ovirt.org/develop/infra/testing/>, using nightly snapshots <http://www.ovirt.org/develop/dev-process/install-nightly-snapshot/>. Switching to Fedora 25 For oVirt 4.2, support for Fedora will be based on Fedora 25. You can help to get it ready by running nightly snapshots <http://www.ovirt.org/develop/dev-process/install-nightly-snapshot/> and reporting any issues. Create Your Own Test Cases We’d love you to contribute some much needed test cases for oVirt features <http://www.ovirt.org/develop/release-management/features/>. Get Creative You are welcome to contribute artwork, such as oVirt live backgrounds and covers, release banners, stickers, and t-shirt designs. Check out Fedora artworks <https://fedoraproject.org/wiki/Artwork#Resources> for inspiration. Get Alternative Into alternative architectures? Test ppc64le <https://cbs.centos.org/repos/virt7-ovirt-41-release/ppc64le/> and aarch64 <https://cbs.centos.org/repos/virt7-ovirt-41-release/aarch64/> packages from CentOS Virt SIG. Talk to Us! Feel free to ask any other question about development by emailing us at devel(a)ovirt.org, or by reaching out on the IRC channel <https://www.ovirt.org/community/>. Spread the Good News Help to spread the word about oVirt! We are the complete virtual machine management system that’s free and comes with an impressive list of enterprise capabilities <http://www.zdnet.com/article/how-much-longer-can-red-hats-ovirt-remain-cove…> including live migration, high availability, load balancing, a self-hosted engine, an intuitive user-interface, and much more. How to Contribute If you’d like to get involved, we’d love to hear from you! Simply shoot us an email at devel(a)ovirt.org and tell us how you’d like to contribute. We look forward to hearing from you!. The oVirt Development Team oVirt: Better technology. Faster Innovation. Powered by community innovation.

1 0

How to deploy hosted-engine to the second and later host machines
by Tatsuya 21 Mar '17

21 Mar '17

Hi, I'm running oVirt 4.1.0.4-1. I'm building oVirt+GlusterFS(replica 3 Volume) configuration for testing. Each machine spec is as follows. CPU: 2core Memory: 8GB Disk: 120GB OS: CentOS7.3 Each host name is host1, host2, host3. I installed Hosted Engine successfully in host1, but I couldn't install it in host2 and host3. In version 4.0, as described below, since it is asked whether or not it is an additional host at the time of deployment, hosted_engine could be added to the second and later node without any error. *** oVirt 4.0 *** [root@host2 ~]# hosted-engine --deploy --config-append=answers.conf ... The specified storage location already contains a data domain. Is this an additional host setup (Yes, No)[Yes]? [ INFO ] Installing on additional host Please specify the Host ID [Must be integer, default: 2]: iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]: No Enter engine admin password: Confirm engine admin password: Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_2]: Enter engine admin password: Confirm engine admin password: Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_2]: [ INFO ] Stage: Setup validation Please provide the fully qualified domain name of this host. Note: The engine VM and all the other hosts should be able to correctly resolve it. Host FQDN: [host2.jitaku.test]: [WARNING] Failed to resolve host2.jitaku.test using DNS, it can be resolved only locally --== CONFIGURATION PREVIEW ==-- Engine FQDN : ovirt-engine.jitaku.test Bridge name : ovirtmgmt Host address : host2.jitaku.test SSH daemon port : 22 Gateway address : 192.168.12.1 Host name for web application : hosted_engine_2 Storage Domain type : glusterfs Host ID : 2 Image size GB : 10 Storage connection : host1.jitaku.test:/engine Console type : vnc Memory size MB : 6144 MAC address : 00:16:3e:XX:XX:XX Boot type : disk Number of CPUs : 2 CPU Type : model_Nehalem But oVirt 4.1 asks for nothing and return error that host1 already has data. *** oVirt 4.1 *** [root@host2 ~]# hosted-engine --deploy --config-append=answers.conf [ INFO ] Stage: Initializing [ INFO ] Generating a temporary VNC password. [ INFO ] Stage: Environment setup During customization use CTRL-D to abort. [ INFO ] Hardware supports virtualization Configuration files: ['/root/answers.conf'] Log file: /var/log/ovirt-hosted-engine-s etup/ovirt-hosted-engine-setup-20170319010957-2gp4od.log Version: otopi-1.6.1 (otopi-1.6.1-1.el7) [ INFO ] Detecting available oVirt engine appliances [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization --== STORAGE CONFIGURATION ==-- [ INFO ] Please note that Replica 3 support is required for the shared storage. [ INFO ] GlusterFS replica 3 Volume detected [ ERROR ] The selected device already contains a storage domain. [ ERROR ] Setup of additional hosts using this software is not allowed anymore. Please use the engine web interface to deploy any additional hosts. [ ERROR ] Failed to execute stage 'Environment customization': Setup of additional hosts using this software is not allowed anymore. Please use the engine web interface to deploy any additional hosts. [ INFO ] Stage: Clean up [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine- setup/answers/answers-20170319011017.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed Log file is located at /var/log/ovirt-hosted-engine-s etup/ovirt-hosted-engine-setup-20170319010957-2gp4od.log I tried it from Virtualization> Hosted Engine> Standard of Cockpit UI, but I received the similar error. How can I deploy hosted-engine on multiple hosts? I'm sorry if I made something misunderstood. Thanks.

3 7

assigned Host was not available
by Bill James 20 Mar '17

20 Mar '17

I have a hardware node that I recently upgraded from 3.6.4 to 4.1.0.4-1. It is having issues that certain VMs I cannot deploy to it because ovirt says host is not available. However that host is status up and has 14 running VMs on it. How do I tell why ovirt thinks the host is not available? Attached is output from "vdsClient -s ovirt4.test.j2noc.com getVdsStats" vdsm-4.19.4-1.el7.centos.x86_64 ovirt-engine-4.1.0.4-1.el7.centos.noarch engine.log: 2017-03-07 11:06:49,269-08 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJo inPool-1-worker-0) [] EVENT_ID: USER_RUN_VM_ON_NON_DEFAULT_VDS(152), Correlation ID: 81490f5b-3029-44d4-b223-df 6bb506efb3, Job ID: 76900dc9-ed9a-4fda-a17a-31825aa515c9, Call Stack: null, Custom Event ID: -1, Message: Guest billj4 started on Host ovirt7.test.j2noc.com. (Default Host parameter was ignored - assigned Host was not available).

2 6

WebAdmin is down, HostedEngine is up
by Logan Kuhn 20 Mar '17

20 Mar '17

Starting at 1:09am on Saturday the Hosted Engine has been rebooting because it failed it's liveliness check. This is due to the webadmin not loading. Nothing changed as far as I can tell on the engine since it's last successful reboot on Friday afternoon. The engine, dwhd and httpd are all up and do not seem to be reporting anything unusual in their respective logs. The engine can talk to the database as I can login using the credentials in /etc/ovirt-engine/engine.co nf.d/10-setup-database.conf and the logs on the postgres server are showing activity. I tried to run engine-setup but it says it's not in global maintenance even though the hosted engine hosts agree that it is. We are on version 4.0.6.3 Server, engine and agent logs are attached Regards, Logan

3 11

Updated ovirt-guest-agent puppet module
by René Koch 20 Mar '17

20 Mar '17

This is a cryptographically signed message in MIME format. --------------ms030208070503090108080000 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Hi, there's an updated version of scrat14/ovirt-guest-agent available at=20 https://forge.puppet.com/scrat14/ovirt_guest_agent Latest version includes support for installing oVirt guest agent on Ubunt= u. Regards, Ren=C3=A9 --------------ms030208070503090108080000 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC CtUwggTrMIID06ADAgECAhAja0pvtE6eFeIAvVog9AvcMA0GCSqGSIb3DQEBCwUAMHUxCzAJ BgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSkwJwYDVQQLEyBTdGFydENvbSBD ZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTEjMCEGA1UEAxMaU3RhcnRDb20gQ2xhc3MgMSBDbGll bnQgQ0EwHhcNMTYwNjA2MTQwNjA0WhcNMTkwOTA2MTQwNjA0WjA4MRcwFQYDVQQDDA5ya29j aEByay1pdC5hdDEdMBsGCSqGSIb3DQEJARYOcmtvY2hAcmstaXQuYXQwggEiMA0GCSqGSIb3 DQEBAQUAA4IBDwAwggEKAoIBAQDTzwY3gPTUDKgjrFFJa0rWfBjMuhDfVnveWxpTRG81IKQ1 4eClq53KsFsegqhf6vTRZ8M+frzEIT4J3DkTrYfYIbK8rN3VIR7TXVPYAA5g9QDJz9CWOCej F7xmw2tOKpFa6ozBPxB8k2Irpl7KTXWmFZ/iN9+wCersY/7hYiXh/S59G7YEioJciaXkdqfR 8R0QDkEcVRAQme2SgjU1tYGJemX+2XYx3Y9z43QgBLdeAwPXYoSdp1TEpOQh+heiKNNm138V Bm8sD/NDxm3ksnhNMjKvqUhVK9z35iksNG7Ao7rgGMtwhd8+mCgFSRQ27lcwwN7cbUpibkGs kQGsoQD/AgMBAAGjggGyMIIBrjAOBgNVHQ8BAf8EBAMCBLAwHQYDVR0lBBYwFAYIKwYBBQUH AwIGCCsGAQUFBwMEMAkGA1UdEwQCMAAwHQYDVR0OBBYEFAov/LAlauAjKdUMLVlx8JW7P63Q MB8GA1UdIwQYMBaAFCSBbDlhvkkPj7cbRivJKLUnSG1oMG8GCCsGAQUFBwEBBGMwYTAkBggr BgEFBQcwAYYYaHR0cDovL29jc3Auc3RhcnRzc2wuY29tMDkGCCsGAQUFBzAChi1odHRwOi8v YWlhLnN0YXJ0c3NsLmNvbS9jZXJ0cy9zY2EuY2xpZW50MS5jcnQwOAYDVR0fBDEwLzAtoCug KYYnaHR0cDovL2NybC5zdGFydHNzbC5jb20vc2NhLWNsaWVudDEuY3JsMBkGA1UdEQQSMBCB DnJrb2NoQHJrLWl0LmF0MCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzBH BgNVHSAEQDA+MDwGCysGAQQBgbU3AQIFMC0wKwYIKwYBBQUHAgEWH2h0dHBzOi8vd3d3LnN0 YXJ0c3NsLmNvbS9wb2xpY3kwDQYJKoZIhvcNAQELBQADggEBAK/kRL1+Hopu1wu7Nni3KEZj CpCDNcUZFgK+TVKEBKLwYgOuy0qe5+z3buQVBJUWlUGKUeY/ifPWU2GKU9X/nxnihQiaOhBW O9XcXg6PcY3N6Vk5akvHE1OWxK0QfpLJYe0jP3PA5fqgWSEUFZQYDz4v3izeSyy+nG7XGC1o mmLEVhAaa2OqaBXphFwfSmdHUmdfrBXwXUI1DMiZX1W13VY/Tw8CWuOLOZ8XWl1IUOTRoO8a XYEUeKjbtEJTaEEUzBpihoN0ueeK5E2wtbI1mfhOoCGpg3EGoKdSV0OR4AQOhzMQRqKMKGfY 4Nq9IjwzwI6TE7Thy5xYjaIzzi/1Ei8wggXiMIIDyqADAgECAhBrp4p9CteI1lEK+Vnk57Th MA0GCSqGSIb3DQEBCwUAMH0xCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQu MSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMSkwJwYDVQQD EyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0xNTEyMTYwMTAwMDVaFw0z MDEyMTYwMTAwMDVaMHUxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSkw JwYDVQQLEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTEjMCEGA1UEAxMaU3Rh cnRDb20gQ2xhc3MgMSBDbGllbnQgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB AQC9fdr3w6J9g/Zbgv3bW1+uHht1wLUZr5gkrLtXedg17AkefMyUGwrQdvwObhajcVmnKVxh rUwkZPXRAwZZosRHfEIi5FH7x6SV/8Sp5lZEuiMnvMFG2MzLA84J6Ws5T4NfXZ0qn4TPgnr3 X2vPVS51M7Ua9nIJgn8jvTra4eyyQzxvuA/GZwKg7VQfDCmCS+kICslYYWgXOMt2xlsSslxL ce0CGWRsT8EpMyt1iDflSjXZIsE7m1uTyHaKZspMLyIyz6mySu8j8BWWHpChNNeTrFuhVfrO AyDPFJVUvKZCLKBhibTLloyy+LatoWELrjdI4a8StZY8+dIR9t4APXGzAgMBAAGjggFkMIIB YDAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMBIGA1Ud EwEB/wQIMAYBAf8CAQAwMgYDVR0fBCswKTAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5j b20vc2ZzY2EuY3JsMGYGCCsGAQUFBwEBBFowWDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3Au c3RhcnRzc2wuY29tMDAGCCsGAQUFBzAChiRodHRwOi8vYWlhLnN0YXJ0c3NsLmNvbS9jZXJ0 cy9jYS5jcnQwHQYDVR0OBBYEFCSBbDlhvkkPj7cbRivJKLUnSG1oMB8GA1UdIwQYMBaAFE4L 7xqkQFulF2mHMMo0aEPQQa7yMD8GA1UdIAQ4MDYwNAYEVR0gADAsMCoGCCsGAQUFBwIBFh5o dHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9wb2xpY3kwDQYJKoZIhvcNAQELBQADggIBAIvj94fs AYuErQ8BAluc4SMnIwS9NPBwAm5SH9uh2NCXTq7im61g7F1LIiNI/+wq37fUuaMbz4g7VarK QTgf8ubs0p7NZWcIe7Bvem2AWaXBsxsaRTYw5kG3DN8pd1hSEUuFoTa7DmNeFe8tiK1BrL3r bA/m48jp4AiFXgvxprJrW7izsyetOrRHPbkW4Y07v29MdhaPv3u1JELyszXqOzjIYo4sWlC8 iDQXwgSW/ntvWy2n4LuiaozlCfXl149tKeqvwlvrla2Yklue/quWp9j9ou4T/OY0CXMuY+B8 wNK0ohd2D4ShgFlMSjzAFRoHGKF81snTr2d1A7Ew02oF6UQyCkC2aNNsK5cWOojBar5c7Hpl X9aHYUCZouxIeU28SONJAxnATgR4cJ2jrpmYSz/kliUJ46S6UpVDo/ebn9c6PaM/XtDYCCaM /7XX6wc3s++sbQ7CtCn1Ax7df6ufQbwyO0V+oFa9H0KAsjHMzcwk3EV2B2NLatidKE/m7G+r B9m+FlVgIiSp0mGlg43QO9Kh1+JqvTCIzv2bJJkmPMLQJNuKKwHNL8F4GGp6jbAV+WL+LDeG fVcq8DHS3LrD+xyYEXQBiqZEdiPVOMxLDSUCXsDO0uCWpaNQ8j6y6S9p0xE/Ga0peVLadVHh qf9nXqKaxnr358VgfrxzUIrvOaOjMYICjzCCAosCAQEwgYkwdTELMAkGA1UEBhMCSUwxFjAU BgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24g QXV0aG9yaXR5MSMwIQYDVQQDExpTdGFydENvbSBDbGFzcyAxIENsaWVudCBDQQIQI2tKb7RO nhXiAL1aIPQL3DANBglghkgBZQMEAgEFAKCB1zAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcB MBwGCSqGSIb3DQEJBTEPFw0xNzAzMjAxMzM4MTRaMC8GCSqGSIb3DQEJBDEiBCDDoK3G/hev KQ+BQL42R4CdwYm0B7M6AzFbKnjjZHT9yjBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQB KjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMC AgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMA0GCSqGSIb3DQEBAQUABIIBAEWPASWjij1f MNyybDSeUDEfOMeC2RU7OmqON79hJrisjwFbafSUuzx9U3ltoWoJXJJ7zlvkdw4fWc4wkd6t 9o21+6uPy7x0TBQ37mFvUkQ2+xwlTBsQsvwYPG9bwrwjZbhFANAX+1/yNajNqdUJziMiUQxN rSGnyhrv4bw++b2dgrUk9umhmqZR+3V0mW98J7PIE54ywZMZFyydIriirgKKg1VLhfCwHZo/ ux2ogh1uNeOa18zAqvfr0KavYcYJIwifPwRaAOlB4k7G5zWAL8PaaUw8eJVTA9S/tHjsvWfu 68vkbiUljL1SfK91UEkiRH97ybgCY0CC/WPljpvzrYkAAAAAAAA= --------------ms030208070503090108080000--

1 0