<html>Hi,<br />I don't beleive that this is relatd to a host, tests have been done from victor source to ginger dest and ginger to victor. I don't see problems on storage (gluster 3.12 native managed by ovirt), when VMs with a uniq disk from 20 to 250G migrate without error in some seconds and with no downtime.<br />How ca I enable this libvirt debug mode ?<br /><br />--<br /><style type="text/css">.Text1 {
color: black;
font-size:9pt;
font-family:Verdana;
}
.Text2 {
color: black;
font-size:7pt;
font-family:Verdana;
}</style><p class="Text1">Cordialement,<br /><br /><b>Frank Soyer </b><br /><br /> </p><br /><br />Le Vendredi, Février 23, 2018 09:56 CET, Milan Zamazal <mzamazal@redhat.com> a écrit:<br /> <blockquote type="cite" cite="87y3jkrsmp.fsf@redhat.com">Maor Lipchuk <mlipchuk@redhat.com> writes:<br /><br />> I encountered a bug (see [1]) which contains the same error mentioned in<br />> your VDSM logs (see [2]), but I doubt it is related.<br /><br />Indeed, it's not related.<br /><br />The error in vdsm_victor.log just means that the info gathering call<br />tries to access libvirt domain before the incoming migration is<br />completed. It's ugly but harmless.<br /><br />> Milan, maybe you have any advice to troubleshoot the issue? Will the<br />> libvirt/qemu logs can help?<br /><br />It seems there is something wrong on (at least) the source host. There<br />are no migration progress messages in the vdsm_ginger.log and there are<br />warnings about stale stat samples. That looks like problems with<br />calling libvirt – slow and/or stuck calls, maybe due to storage<br />problems. The possibly faulty second disk could cause that.<br /><br />libvirt debug logs could tell us whether that is indeed the problem and<br />whether it is caused by storage or something else.<br /><br />> I would suggest to open a bug on that issue so we can track it more<br />> properly.<br />><br />> Regards,<br />> Maor<br />><br />><br />> [1]<br />> https://bugzilla.redhat.com/show_bug.cgi?id=1486543 - Migration leads to<br />> VM running on 2 Hosts<br />><br />> [2]<br />> 2018-02-16 09:43:35,236+0100 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer]<br />> Internal server error (__init__:577)<br />> Traceback (most recent call last):<br />> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 572,<br />> in _handle_request<br />> res = method(**params)<br />> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in<br />> _dynamicMethod<br />> result = fn(*methodArgs)<br />> File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies<br />> io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()<br />> File "/usr/share/vdsm/clientIF.py", line 454, in getAllVmIoTunePolicies<br />> 'current_values': v.getIoTune()}<br />> File "/usr/share/vdsm/virt/vm.py", line 2859, in getIoTune<br />> result = self.getIoTuneResponse()<br />> File "/usr/share/vdsm/virt/vm.py", line 2878, in getIoTuneResponse<br />> res = self._dom.blockIoTune(<br />> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47,<br />> in __getattr__<br />> % self.vmid)<br />> NotConnectedError: VM u'755cf168-de65-42ed-b22f-efe9136f7594' was not<br />> started yet or was shut down<br />><br />> On Thu, Feb 22, 2018 at 4:22 PM, fsoyer <fsoyer@systea.fr> wrote:<br />><br />>> Hi,<br />>> Yes, on 2018-02-16 (vdsm logs) I tried with a VM standing on ginger<br />>> (192.168.0.6) migrated (or failed to migrate...) to victor (192.168.0.5),<br />>> while the engine.log in the first mail on 2018-02-12 was for VMs standing<br />>> on victor, migrated (or failed to migrate...) to ginger. Symptoms were<br />>> exactly the same, in both directions, and VMs works like a charm before,<br />>> and even after (migration "killed" by a poweroff of VMs).<br />>> Am I the only one experimenting this problem ?<br />>><br />>><br />>> Thanks<br />>> --<br />>><br />>> Cordialement,<br />>><br />>> *Frank Soyer *<br />>><br />>><br />>><br />>> Le Jeudi, Février 22, 2018 00:45 CET, Maor Lipchuk <mlipchuk@redhat.com><br />>> a écrit:<br />>><br />>><br />>> Hi Frank,<br />>><br />>> Sorry about the delay repond.<br />>> I've been going through the logs you attached, although I could not find<br />>> any specific indication why the migration failed because of the disk you<br />>> were mentionning.<br />>> Does this VM run with both disks on the target host without migration?<br />>><br />>> Regards,<br />>> Maor<br />>><br />>><br />>> On Fri, Feb 16, 2018 at 11:03 AM, fsoyer <fsoyer@systea.fr> wrote:<br />>>><br />>>> Hi Maor,<br />>>> sorry for the double post, I've change the email adress of my account and<br />>>> supposed that I'd need to re-post it.<br />>>> And thank you for your time. Here are the logs. I added a vdisk to an<br />>>> existing VM : it no more migrates, needing to poweroff it after minutes.<br />>>> Then simply deleting the second disk makes migrate it in exactly 9s without<br />>>> problem !<br />>>> https://gist.github.com/fgth/4707446331d201eef574ac31b6e89561<br />>>> https://gist.github.com/fgth/f8de9c22664aee53722af676bff8719d<br />>>><br />>>> --<br />>>><br />>>> Cordialement,<br />>>><br />>>> *Frank Soyer *<br />>>> Le Mercredi, Février 14, 2018 11:04 CET, Maor Lipchuk <<br />>>> mlipchuk@redhat.com> a écrit:<br />>>><br />>>><br />>>> Hi Frank,<br />>>><br />>>> I already replied on your last email.<br />>>> Can you provide the VDSM logs from the time of the migration failure for<br />>>> both hosts:<br />>>> ginger.local.systea.f <http://ginger.local.systea.fr/>r and v<br />>>> ictor.local.systea.fr<br />>>><br />>>> Thanks,<br />>>> Maor<br />>>><br />>>> On Wed, Feb 14, 2018 at 11:23 AM, fsoyer <fsoyer@systea.fr> wrote:<br />>>>><br />>>>> Hi all,<br />>>>> I discovered yesterday a problem when migrating VM with more than one<br />>>>> vdisk.<br />>>>> On our test servers (oVirt4.1, shared storage with Gluster), I created 2<br />>>>> VMs needed for a test, from a template with a 20G vdisk. On this VMs I<br />>>>> added a 100G vdisk (for this tests I didn't want to waste time to extend<br />>>>> the existing vdisks... But I lost time finally...). The VMs with the 2<br />>>>> vdisks works well.<br />>>>> Now I saw some updates waiting on the host. I tried to put it in<br />>>>> maintenance... But it stopped on the two VM. They were marked "migrating",<br />>>>> but no more accessible. Other (small) VMs with only 1 vdisk was migrated<br />>>>> without problem at the same time.<br />>>>> I saw that a kvm process for the (big) VMs was launched on the source<br />>>>> AND destination host, but after tens of minutes, the migration and the VMs<br />>>>> was always freezed. I tried to cancel the migration for the VMs : failed.<br />>>>> The only way to stop it was to poweroff the VMs : the kvm process died on<br />>>>> the 2 hosts and the GUI alerted on a failed migration.<br />>>>> In doubt, I tried to delete the second vdisk on one of this VMs : it<br />>>>> migrates then without error ! And no access problem.<br />>>>> I tried to extend the first vdisk of the second VM, the delete the<br />>>>> second vdisk : it migrates now without problem !<br />>>>><br />>>>> So after another test with a VM with 2 vdisks, I can say that this<br />>>>> blocked the migration process :(<br />>>>><br />>>>> In engine.log, for a VMs with 1 vdisk migrating well, we see :<br />>>>><br />>>>> 2018-02-12 16:46:29,705+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand]<br />>>>> (default task-28) [2f712024-5982-46a8-82c8-fd8293da5725] Lock Acquired<br />>>>> to object 'EngineLock:{exclusiveLocks='[3f57e669-5e4c-4d10-85cc-d573004a099d=VM]',<br />>>>> sharedLocks=''}'<br />>>>> 2018-02-12 16:46:29,955+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand]<br />>>>> (org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725]<br />>>>> Running command: MigrateVmToServerCommand internal: false. Entities<br />>>>> affected : ID: 3f57e669-5e4c-4d10-85cc-d573004a099d Type: VMAction<br />>>>> group MIGRATE_VM with role type USER<br />>>>> 2018-02-12 16:46:30,261+01 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]<br />>>>> (org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725]<br />>>>> START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true',<br />>>>> hostId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d', srcHost='192.168.0.6',<br />>>>> dstVdsId='d569c2dd-8f30-4878-8aea-858db285cf69', dstHost='<br />>>>> 192.168.0.5:54321', migrationMethod='ONLINE', tunnelMigration='false',<br />>>>> migrationDowntime='0', autoConverge='true', migrateCompressed='false',<br />>>>> consoleAddress='null', maxBandwidth='500', enableGuestEvents='true',<br />>>>> maxIncomingMigrations='2', maxOutgoingMigrations='2',<br />>>>> convergenceSchedule='[init=[{name=setDowntime, params=[100]}],<br />>>>> stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,<br />>>>> action={name=setDowntime, params=[200]}}, {limit=3,<br />>>>> action={name=setDowntime, params=[300]}}, {limit=4,<br />>>>> action={name=setDowntime, params=[400]}}, {limit=6,<br />>>>> action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort,<br />>>>> params=[]}}]]'}), log id: 14f61ee0<br />>>>> 2018-02-12 16:46:30,262+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-32)<br />>>>> [2f712024-5982-46a8-82c8-fd8293da5725] START,<br />>>>> MigrateBrokerVDSCommand(HostName = victor.local.systea.fr,<br />>>>> MigrateVDSCommandParameters:{runAsync='true',<br />>>>> hostId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d', srcHost='192.168.0.6',<br />>>>> dstVdsId='d569c2dd-8f30-4878-8aea-858db285cf69', dstHost='<br />>>>> 192.168.0.5:54321', migrationMethod='ONLINE', tunnelMigration='false',<br />>>>> migrationDowntime='0', autoConverge='true', migrateCompressed='false',<br />>>>> consoleAddress='null', maxBandwidth='500', enableGuestEvents='true',<br />>>>> maxIncomingMigrations='2', maxOutgoingMigrations='2',<br />>>>> convergenceSchedule='[init=[{name=setDowntime, params=[100]}],<br />>>>> stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,<br />>>>> action={name=setDowntime, params=[200]}}, {limit=3,<br />>>>> action={name=setDowntime, params=[300]}}, {limit=4,<br />>>>> action={name=setDowntime, params=[400]}}, {limit=6,<br />>>>> action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort,<br />>>>> params=[]}}]]'}), log id: 775cd381<br />>>>> 2018-02-12 16:46:30,277+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-32)<br />>>>> [2f712024-5982-46a8-82c8-fd8293da5725] FINISH, MigrateBrokerVDSCommand,<br />>>>> log id: 775cd381<br />>>>> 2018-02-12 16:46:30,285+01 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]<br />>>>> (org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725]<br />>>>> FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 14f61ee0<br />>>>> 2018-02-12 16:46:30,301+01 INFO [org.ovirt.engine.core.dal.db<br />>>>> broker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32)<br />>>>> [2f712024-5982-46a8-82c8-fd8293da5725] EVENT_ID:<br />>>>> VM_MIGRATION_START(62), Correlation ID: 2f712024-5982-46a8-82c8-fd8293da5725,<br />>>>> Job ID: 4bd19aa9-cc99-4d02-884e-5a1e857a7738, Call Stack: null, Custom<br />>>>> ID: null, Custom Event ID: -1, Message: Migration started (VM:<br />>>>> Oracle_SECONDARY, Source: victor.local.systea.fr, Destination:<br />>>>> ginger.local.systea.fr, User: admin@internal-authz).<br />>>>> 2018-02-12 16:46:31,106+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler9) [54a65b66]<br />>>>> START, FullListVDSCommand(HostName = victor.local.systea.fr,<br />>>>> FullListVDSCommandParameters:{runAsync='true',<br />>>>> hostId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1',<br />>>>> vmIds='[3f57e669-5e4c-4d10-85cc-d573004a099d]'}), log id: 54b4b435<br />>>>> 2018-02-12 16:46:31,147+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler9) [54a65b66]<br />>>>> FINISH, FullListVDSCommand, return: [{acpiEnable=true,<br />>>>> emulatedMachine=pc-i440fx-rhel7.3.0, tabletEnable=true, pid=1493,<br />>>>> guestDiskMapping={0QEMU_QEMU_HARDDISK_d890fa68-fba4-4f49-9={name=/dev/sda},<br />>>>> QEMU_DVD-ROM_QM00003={name=/dev/sr0}}, transparentHugePages=true,<br />>>>> timeOffset=0, cpuType=Nehalem, smp=2, pauseCode=NOERR,<br />>>>> guestNumaNodes=[Ljava.lang.Object;@1d9042cd, smartcardEnable=false,<br />>>>> custom={device_fbddd528-7d93-49c6-a286-180e021cb274device_87<br />>>>> 9c93ab-4df1-435c-af02-565039fcc254=VmDevice:{id='VmDeviceId:<br />>>>> {deviceId='879c93ab-4df1-435c-af02-565039fcc254',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}', device='unix',<br />>>>> type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0,<br />>>>> controller=0, type=virtio-serial, port=1}', managed='false',<br />>>>> plugged='true', readOnly='false', deviceAlias='channel0',<br />>>>> customProperties='[]', snapshotId='null', logicalName='null',<br />>>>> hostDevice='null'}, device_fbddd528-7d93-49c6-a286<br />>>>> -180e021cb274device_879c93ab-4df1-435c-af02-565039fcc254devi<br />>>>> ce_8945f61a-abbe-4156-8485-a4aa6f1908dbdevice_017b5e59-01c4-<br />>>>> 4aac-bf0c-b5d9557284d6=VmDevice:{id='VmDeviceId:{deviceId='0<br />>>>> 17b5e59-01c4-4aac-bf0c-b5d9557284d6', vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}',<br />>>>> device='tablet', type='UNKNOWN', bootOrder='0', specParams='[]',<br />>>>> address='{bus=0, type=usb, port=1}', managed='false', plugged='true',<br />>>>> readOnly='false', deviceAlias='input0', customProperties='[]',<br />>>>> snapshotId='null', logicalName='null', hostDevice='null'},<br />>>>> device_fbddd528-7d93-49c6-a286-180e021cb274=VmDevice:{id='Vm<br />>>>> DeviceId:{deviceId='fbddd528-7d93-49c6-a286-180e021cb274',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}', device='ide',<br />>>>> type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01,<br />>>>> bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false',<br />>>>> plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]',<br />>>>> snapshotId='null', logicalName='null', hostDevice='null'},<br />>>>> device_fbddd528-7d93-49c6-a286-180e021cb274device_879c93ab-4<br />>>>> df1-435c-af02-565039fcc254device_8945f61a-abbe-4156-8485-a4a<br />>>>> a6f1908db=VmDevice:{id='VmDeviceId:{deviceId='8945f61a-abbe-4156-8485-a4aa6f1908db',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}', device='unix',<br />>>>> type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0,<br />>>>> controller=0, type=virtio-serial, port=2}', managed='false',<br />>>>> plugged='true', readOnly='false', deviceAlias='channel1',<br />>>>> customProperties='[]', snapshotId='null', logicalName='null',<br />>>>> hostDevice='null'}}, vmType=kvm, memSize=8192, smpCoresPerSocket=1,<br />>>>> vmName=Oracle_SECONDARY, nice=0, status=Migration Source, maxMemSize=32768,<br />>>>> bootMenuEnable=false, vmId=3f57e669-5e4c-4d10-85cc-d573004a099d,<br />>>>> numOfIoThreads=2, smpThreadsPerCore=1, memGuaranteedSize=8192,<br />>>>> kvmEnable=true, pitReinjection=false, displayNetwork=ovirtmgmt,<br />>>>> devices=[Ljava.lang.Object;@28ae66d7, display=vnc, maxVCpus=16,<br />>>>> clientIp=, statusTime=4299484520, maxMemSlots=16}], log id: 54b4b435<br />>>>> 2018-02-12 16:46:31,150+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler1)<br />>>>> [27fac647] Fetched 3 VMs from VDS 'd569c2dd-8f30-4878-8aea-858db285cf69'<br />>>>> 2018-02-12 16:46:31,151+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9)<br />>>>> [54a65b66] Received a vnc Device without an address when processing VM<br />>>>> 3f57e669-5e4c-4d10-85cc-d573004a099d devices, skipping device:<br />>>>> {device=vnc, specParams={displayNetwork=ovirtmgmt, keyMap=fr,<br />>>>> displayIp=192.168.0.6}, type=graphics, deviceId=813957b1-446a-4e88-9e40-9fe76d2c442d,<br />>>>> port=5901}<br />>>>> 2018-02-12 16:46:31,151+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9)<br />>>>> [54a65b66] Received a lease Device without an address when processing VM<br />>>>> 3f57e669-5e4c-4d10-85cc-d573004a099d devices, skipping device:<br />>>>> {lease_id=3f57e669-5e4c-4d10-85cc-d573004a099d,<br />>>>> sd_id=1e51cecc-eb2e-47d0-b185-920fdc7afa16,<br />>>>> deviceId={uuid=a09949aa-5642-4b6d-94a4-8b0d04257be5}, offset=6291456,<br />>>>> device=lease, path=/rhev/data-center/mnt/glusterSD/192.168.0.6:<br />>>>> _DATA01/1e51cecc-eb2e-47d0-b185-920fdc7afa16/dom_md/xleases, type=lease}<br />>>>> 2018-02-12 16:46:31,152+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]<br />>>>> (DefaultQuartzScheduler1) [27fac647] VM '3f57e669-5e4c-4d10-85cc-d573004a099d'(Oracle_SECONDARY)<br />>>>> was unexpectedly detected as 'MigratingTo' on VDS<br />>>>> 'd569c2dd-8f30-4878-8aea-858db285cf69'(ginger.local.systea.fr)<br />>>>> (expected on 'ce3938b1-b23f-4d22-840a-f17d7cd87bb1')<br />>>>> 2018-02-12 16:46:31,152+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]<br />>>>> (DefaultQuartzScheduler1) [27fac647] VM '3f57e669-5e4c-4d10-85cc-d573004a099d'<br />>>>> is migrating to VDS 'd569c2dd-8f30-4878-8aea-858db285cf69'(<br />>>>> ginger.local.systea.fr) ignoring it in the refresh until migration is<br />>>>> done<br />>>>> ....<br />>>>> 2018-02-12 16:46:41,631+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]<br />>>>> (ForkJoinPool-1-worker-11) [] VM '3f57e669-5e4c-4d10-85cc-d573004a099d'<br />>>>> was reported as Down on VDS 'ce3938b1-b23f-4d22-840a-f17d7cd87bb1'(<br />>>>> victor.local.systea.fr)<br />>>>> 2018-02-12 16:46:41,632+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-11) [] START,<br />>>>> DestroyVDSCommand(HostName = victor.local.systea.fr,<br />>>>> DestroyVmVDSCommandParameters:{runAsync='true',<br />>>>> hostId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d', force='false',<br />>>>> secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log<br />>>>> id: 560eca57<br />>>>> 2018-02-12 16:46:41,650+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-11) [] FINISH,<br />>>>> DestroyVDSCommand, log id: 560eca57<br />>>>> 2018-02-12 16:46:41,650+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]<br />>>>> (ForkJoinPool-1-worker-11) [] VM '3f57e669-5e4c-4d10-85cc-d573004a099d'(Oracle_SECONDARY)<br />>>>> moved from 'MigratingFrom' --> 'Down'<br />>>>> 2018-02-12 16:46:41,651+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]<br />>>>> (ForkJoinPool-1-worker-11) [] Handing over VM '3f57e669-5e4c-4d10-85cc-d573004a099d'(Oracle_SECONDARY)<br />>>>> to Host 'd569c2dd-8f30-4878-8aea-858db285cf69'. Setting VM to status<br />>>>> 'MigratingTo'<br />>>>> 2018-02-12 16:46:42,163+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]<br />>>>> (ForkJoinPool-1-worker-4) [] VM '3f57e669-5e4c-4d10-85cc-d573004a099d'(Oracle_SECONDARY)<br />>>>> moved from 'MigratingTo' --> 'Up'<br />>>>> 2018-02-12 16:46:42,169+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.MigrateStatusVDSCommand] (ForkJoinPool-1-worker-4) []<br />>>>> START, MigrateStatusVDSCommand(HostName = ginger.local.systea.fr,<br />>>>> MigrateStatusVDSCommandParameters:{runAsync='true',<br />>>>> hostId='d569c2dd-8f30-4878-8aea-858db285cf69',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}), log id: 7a25c281<br />>>>> 2018-02-12 16:46:42,174+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.MigrateStatusVDSCommand] (ForkJoinPool-1-worker-4) []<br />>>>> FINISH, MigrateStatusVDSCommand, log id: 7a25c281<br />>>>> 2018-02-12 16:46:42,194+01 INFO [org.ovirt.engine.core.dal.db<br />>>>> broker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-4) []<br />>>>> EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID:<br />>>>> 2f712024-5982-46a8-82c8-fd8293da5725, Job ID:<br />>>>> 4bd19aa9-cc99-4d02-884e-5a1e857a7738, Call Stack: null, Custom ID:<br />>>>> null, Custom Event ID: -1, Message: Migration completed (VM:<br />>>>> Oracle_SECONDARY, Source: victor.local.systea.fr, Destination:<br />>>>> ginger.local.systea.fr, Duration: 11 seconds, Total: 11 seconds, Actual<br />>>>> downtime: (N/A))<br />>>>> 2018-02-12 16:46:42,201+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand]<br />>>>> (ForkJoinPool-1-worker-4) [] Lock freed to object<br />>>>> 'EngineLock:{exclusiveLocks='[3f57e669-5e4c-4d10-85cc-d573004a099d=VM]',<br />>>>> sharedLocks=''}'<br />>>>> 2018-02-12 16:46:42,203+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-4) [] START,<br />>>>> FullListVDSCommand(HostName = ginger.local.systea.fr,<br />>>>> FullListVDSCommandParameters:{runAsync='true',<br />>>>> hostId='d569c2dd-8f30-4878-8aea-858db285cf69',<br />>>>> vmIds='[3f57e669-5e4c-4d10-85cc-d573004a099d]'}), log id: 7cc65298<br />>>>> 2018-02-12 16:46:42,254+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-4) [] FINISH,<br />>>>> FullListVDSCommand, return: [{acpiEnable=true,<br />>>>> emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=,<br />>>>> tabletEnable=true, pid=18748, guestDiskMapping={},<br />>>>> transparentHugePages=true, timeOffset=0, cpuType=Nehalem, smp=2,<br />>>>> guestNumaNodes=[Ljava.lang.Object;@760085fd,<br />>>>> custom={device_fbddd528-7d93-49c6-a286-180e021cb274device_87<br />>>>> 9c93ab-4df1-435c-af02-565039fcc254=VmDevice:{id='VmDeviceId:<br />>>>> {deviceId='879c93ab-4df1-435c-af02-565039fcc254',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}', device='unix',<br />>>>> type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0,<br />>>>> controller=0, type=virtio-serial, port=1}', managed='false',<br />>>>> plugged='true', readOnly='false', deviceAlias='channel0',<br />>>>> customProperties='[]', snapshotId='null', logicalName='null',<br />>>>> hostDevice='null'}, device_fbddd528-7d93-49c6-a286<br />>>>> -180e021cb274device_879c93ab-4df1-435c-af02-565039fcc254devi<br />>>>> ce_8945f61a-abbe-4156-8485-a4aa6f1908dbdevice_017b5e59-01c4-<br />>>>> 4aac-bf0c-b5d9557284d6=VmDevice:{id='VmDeviceId:{deviceId='0<br />>>>> 17b5e59-01c4-4aac-bf0c-b5d9557284d6', vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}',<br />>>>> device='tablet', type='UNKNOWN', bootOrder='0', specParams='[]',<br />>>>> address='{bus=0, type=usb, port=1}', managed='false', plugged='true',<br />>>>> readOnly='false', deviceAlias='input0', customProperties='[]',<br />>>>> snapshotId='null', logicalName='null', hostDevice='null'},<br />>>>> device_fbddd528-7d93-49c6-a286-180e021cb274=VmDevice:{id='Vm<br />>>>> DeviceId:{deviceId='fbddd528-7d93-49c6-a286-180e021cb274',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}', device='ide',<br />>>>> type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01,<br />>>>> bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false',<br />>>>> plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]',<br />>>>> snapshotId='null', logicalName='null', hostDevice='null'},<br />>>>> device_fbddd528-7d93-49c6-a286-180e021cb274device_879c93ab-4<br />>>>> df1-435c-af02-565039fcc254device_8945f61a-abbe-4156-8485-a4a<br />>>>> a6f1908db=VmDevice:{id='VmDeviceId:{deviceId='8945f61a-abbe-4156-8485-a4aa6f1908db',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}', device='unix',<br />>>>> type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0,<br />>>>> controller=0, type=virtio-serial, port=2}', managed='false',<br />>>>> plugged='true', readOnly='false', deviceAlias='channel1',<br />>>>> customProperties='[]', snapshotId='null', logicalName='null',<br />>>>> hostDevice='null'}}, vmType=kvm, memSize=8192, smpCoresPerSocket=1,<br />>>>> vmName=Oracle_SECONDARY, nice=0, status=Up, maxMemSize=32768,<br />>>>> bootMenuEnable=false, vmId=3f57e669-5e4c-4d10-85cc-d573004a099d,<br />>>>> numOfIoThreads=2, smpThreadsPerCore=1, smartcardEnable=false,<br />>>>> maxMemSlots=16, kvmEnable=true, pitReinjection=false,<br />>>>> displayNetwork=ovirtmgmt, devices=[Ljava.lang.Object;@2e4d3dd3,<br />>>>> memGuaranteedSize=8192, maxVCpus=16, clientIp=, statusTime=4304259600<br />>>>> <(430)%20425-9600>, display=vnc}], log id: 7cc65298<br />>>>> 2018-02-12 16:46:42,257+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-4) []<br />>>>> Received a vnc Device without an address when processing VM<br />>>>> 3f57e669-5e4c-4d10-85cc-d573004a099d devices, skipping device:<br />>>>> {device=vnc, specParams={displayNetwork=ovirtmgmt, keyMap=fr,<br />>>>> displayIp=192.168.0.5}, type=graphics, deviceId=813957b1-446a-4e88-9e40-9fe76d2c442d,<br />>>>> port=5901}<br />>>>> 2018-02-12 16:46:42,257+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-4) []<br />>>>> Received a lease Device without an address when processing VM<br />>>>> 3f57e669-5e4c-4d10-85cc-d573004a099d devices, skipping device:<br />>>>> {lease_id=3f57e669-5e4c-4d10-85cc-d573004a099d,<br />>>>> sd_id=1e51cecc-eb2e-47d0-b185-920fdc7afa16,<br />>>>> deviceId={uuid=a09949aa-5642-4b6d-94a4-8b0d04257be5}, offset=6291456,<br />>>>> device=lease, path=/rhev/data-center/mnt/glusterSD/192.168.0.6:<br />>>>> _DATA01/1e51cecc-eb2e-47d0-b185-920fdc7afa16/dom_md/xleases, type=lease}<br />>>>> 2018-02-12 16:46:46,260+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler5) [7fcb200a]<br />>>>> FINISH, FullListVDSCommand, return: [{acpiEnable=true,<br />>>>> emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=,<br />>>>> tabletEnable=true, pid=18748, guestDiskMapping={0QEMU_QEMU_H<br />>>>> ARDDISK_d890fa68-fba4-4f49-9={name=/dev/sda},<br />>>>> QEMU_DVD-ROM_QM00003={name=/dev/sr0}}, transparentHugePages=true,<br />>>>> timeOffset=0, cpuType=Nehalem, smp=2, guestNumaNodes=[Ljava.lang.Obj<br />>>>> ect;@77951faf, custom={device_fbddd528-7d93-4<br />>>>> 9c6-a286-180e021cb274device_879c93ab-4df1-435c-af02-565039fc<br />>>>> c254=VmDevice:{id='VmDeviceId:{deviceId='879c93ab-4df1-435c-af02-565039fcc254',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}', device='unix',<br />>>>> type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0,<br />>>>> controller=0, type=virtio-serial, port=1}', managed='false',<br />>>>> plugged='true', readOnly='false', deviceAlias='channel0',<br />>>>> customProperties='[]', snapshotId='null', logicalName='null',<br />>>>> hostDevice='null'}, device_fbddd528-7d93-49c6-a286<br />>>>> -180e021cb274device_879c93ab-4df1-435c-af02-565039fcc254devi<br />>>>> ce_8945f61a-abbe-4156-8485-a4aa6f1908dbdevice_017b5e59-01c4-<br />>>>> 4aac-bf0c-b5d9557284d6=VmDevice:{id='VmDeviceId:{deviceId='0<br />>>>> 17b5e59-01c4-4aac-bf0c-b5d9557284d6', vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}',<br />>>>> device='tablet', type='UNKNOWN', bootOrder='0', specParams='[]',<br />>>>> address='{bus=0, type=usb, port=1}', managed='false', plugged='true',<br />>>>> readOnly='false', deviceAlias='input0', customProperties='[]',<br />>>>> snapshotId='null', logicalName='null', hostDevice='null'},<br />>>>> device_fbddd528-7d93-49c6-a286-180e021cb274=VmDevice:{id='Vm<br />>>>> DeviceId:{deviceId='fbddd528-7d93-49c6-a286-180e021cb274',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}', device='ide',<br />>>>> type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01,<br />>>>> bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false',<br />>>>> plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]',<br />>>>> snapshotId='null', logicalName='null', hostDevice='null'},<br />>>>> device_fbddd528-7d93-49c6-a286-180e021cb274device_879c93ab-4<br />>>>> df1-435c-af02-565039fcc254device_8945f61a-abbe-4156-8485-a4a<br />>>>> a6f1908db=VmDevice:{id='VmDeviceId:{deviceId='8945f61a-abbe-4156-8485-a4aa6f1908db',<br />>>>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d'}', device='unix',<br />>>>> type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0,<br />>>>> controller=0, type=virtio-serial, port=2}', managed='false',<br />>>>> plugged='true', readOnly='false', deviceAlias='channel1',<br />>>>> customProperties='[]', snapshotId='null', logicalName='null',<br />>>>> hostDevice='null'}}, vmType=kvm, memSize=8192, smpCoresPerSocket=1,<br />>>>> vmName=Oracle_SECONDARY, nice=0, status=Up, maxMemSize=32768,<br />>>>> bootMenuEnable=false, vmId=3f57e669-5e4c-4d10-85cc-d573004a099d,<br />>>>> numOfIoThreads=2, smpThreadsPerCore=1, smartcardEnable=false,<br />>>>> maxMemSlots=16, kvmEnable=true, pitReinjection=false,<br />>>>> displayNetwork=ovirtmgmt, devices=[Ljava.lang.Object;@286410fd,<br />>>>> memGuaranteedSize=8192, maxVCpus=16, clientIp=, statusTime=4304263620<br />>>>> <(430)%20426-3620>, display=vnc}], log id: 58cdef4c<br />>>>> 2018-02-12 16:46:46,267+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5)<br />>>>> [7fcb200a] Received a vnc Device without an address when processing VM<br />>>>> 3f57e669-5e4c-4d10-85cc-d573004a099d devices, skipping device:<br />>>>> {device=vnc, specParams={displayNetwork=ovirtmgmt, keyMap=fr,<br />>>>> displayIp=192.168.0.5}, type=graphics, deviceId=813957b1-446a-4e88-9e40-9fe76d2c442d,<br />>>>> port=5901}<br />>>>> 2018-02-12 16:46:46,268+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5)<br />>>>> [7fcb200a] Received a lease Device without an address when processing VM<br />>>>> 3f57e669-5e4c-4d10-85cc-d573004a099d devices, skipping device:<br />>>>> {lease_id=3f57e669-5e4c-4d10-85cc-d573004a099d,<br />>>>> sd_id=1e51cecc-eb2e-47d0-b185-920fdc7afa16,<br />>>>> deviceId={uuid=a09949aa-5642-4b6d-94a4-8b0d04257be5}, offset=6291456,<br />>>>> device=lease, path=/rhev/data-center/mnt/glusterSD/192.168.0.6:<br />>>>> _DATA01/1e51cecc-eb2e-47d0-b185-920fdc7afa16/dom_md/xleases, type=lease}<br />>>>><br />>>>><br />>>>><br />>>>><br />>>>> For the VM with 2 vdisks we see :<br />>>>><br />>>>> 2018-02-12 16:49:06,112+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand]<br />>>>> (default task-50) [92b5af33-cb87-4142-b8fe-8b838dd7458e] Lock Acquired<br />>>>> to object 'EngineLock:{exclusiveLocks='[f7d4ec12-627a-4b83-b59e-886400d55474=VM]',<br />>>>> sharedLocks=''}'<br />>>>> 2018-02-12 16:49:06,407+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand]<br />>>>> (org.ovirt.thread.pool-6-thread-49) [92b5af33-cb87-4142-b8fe-8b838dd7458e]<br />>>>> Running command: MigrateVmToServerCommand internal: false. Entities<br />>>>> affected : ID: f7d4ec12-627a-4b83-b59e-886400d55474 Type: VMAction<br />>>>> group MIGRATE_VM with role type USER<br />>>>> 2018-02-12 16:49:06,712+01 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]<br />>>>> (org.ovirt.thread.pool-6-thread-49) [92b5af33-cb87-4142-b8fe-8b838dd7458e]<br />>>>> START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true',<br />>>>> hostId='d569c2dd-8f30-4878-8aea-858db285cf69',<br />>>>> vmId='f7d4ec12-627a-4b83-b59e-886400d55474', srcHost='192.168.0.5',<br />>>>> dstVdsId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1', dstHost='<br />>>>> 192.168.0.6:54321', migrationMethod='ONLINE', tunnelMigration='false',<br />>>>> migrationDowntime='0', autoConverge='true', migrateCompressed='false',<br />>>>> consoleAddress='null', maxBandwidth='500', enableGuestEvents='true',<br />>>>> maxIncomingMigrations='2', maxOutgoingMigrations='2',<br />>>>> convergenceSchedule='[init=[{name=setDowntime, params=[100]}],<br />>>>> stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,<br />>>>> action={name=setDowntime, params=[200]}}, {limit=3,<br />>>>> action={name=setDowntime, params=[300]}}, {limit=4,<br />>>>> action={name=setDowntime, params=[400]}}, {limit=6,<br />>>>> action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort,<br />>>>> params=[]}}]]'}), log id: 3702a9e0<br />>>>> 2018-02-12 16:49:06,713+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-49)<br />>>>> [92b5af33-cb87-4142-b8fe-8b838dd7458e] START,<br />>>>> MigrateBrokerVDSCommand(HostName = ginger.local.systea.fr,<br />>>>> MigrateVDSCommandParameters:{runAsync='true',<br />>>>> hostId='d569c2dd-8f30-4878-8aea-858db285cf69',<br />>>>> vmId='f7d4ec12-627a-4b83-b59e-886400d55474', srcHost='192.168.0.5',<br />>>>> dstVdsId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1', dstHost='<br />>>>> 192.168.0.6:54321', migrationMethod='ONLINE', tunnelMigration='false',<br />>>>> migrationDowntime='0', autoConverge='true', migrateCompressed='false',<br />>>>> consoleAddress='null', maxBandwidth='500', enableGuestEvents='true',<br />>>>> maxIncomingMigrations='2', maxOutgoingMigrations='2',<br />>>>> convergenceSchedule='[init=[{name=setDowntime, params=[100]}],<br />>>>> stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,<br />>>>> action={name=setDowntime, params=[200]}}, {limit=3,<br />>>>> action={name=setDowntime, params=[300]}}, {limit=4,<br />>>>> action={name=setDowntime, params=[400]}}, {limit=6,<br />>>>> action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort,<br />>>>> params=[]}}]]'}), log id: 1840069c<br />>>>> 2018-02-12 16:49:06,724+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-49)<br />>>>> [92b5af33-cb87-4142-b8fe-8b838dd7458e] FINISH, MigrateBrokerVDSCommand,<br />>>>> log id: 1840069c<br />>>>> 2018-02-12 16:49:06,732+01 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]<br />>>>> (org.ovirt.thread.pool-6-thread-49) [92b5af33-cb87-4142-b8fe-8b838dd7458e]<br />>>>> FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 3702a9e0<br />>>>> 2018-02-12 16:49:06,753+01 INFO [org.ovirt.engine.core.dal.db<br />>>>> broker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49)<br />>>>> [92b5af33-cb87-4142-b8fe-8b838dd7458e] EVENT_ID:<br />>>>> VM_MIGRATION_START(62), Correlation ID: 92b5af33-cb87-4142-b8fe-8b838dd7458e,<br />>>>> Job ID: f4f54054-f7c8-4481-8eda-d5a15c383061, Call Stack: null, Custom<br />>>>> ID: null, Custom Event ID: -1, Message: Migration started (VM:<br />>>>> Oracle_PRIMARY, Source: ginger.local.systea.fr, Destination:<br />>>>> victor.local.systea.fr, User: admin@internal-authz).<br />>>>> ...<br />>>>> 2018-02-12 16:49:16,453+01 INFO [org.ovirt.engine.core.vdsbro<br />>>>> ker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler4)<br />>>>> [162a5bc3] Fetched 2 VMs from VDS 'ce3938b1-b23f-4d22-840a-f17d7cd87bb1'<br />>>>> 2018-02-12 16:49:16,455+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]<br />>>>> (DefaultQuartzScheduler4) [162a5bc3] VM 'f7d4ec12-627a-4b83-b59e-886400d55474'(Oracle_PRIMARY)<br />>>>> was unexpectedly detected as 'MigratingTo' on VDS<br />>>>> 'ce3938b1-b23f-4d22-840a-f17d7cd87bb1'(victor.local.systea.fr)<br />>>>> (expected on 'd569c2dd-8f30-4878-8aea-858db285cf69')<br />>>>> 2018-02-12 16:49:16,455+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]<br />>>>> (DefaultQuartzScheduler4) [162a5bc3] VM 'f7d4ec12-627a-4b83-b59e-886400d55474'<br />>>>> is migrating to VDS 'ce3938b1-b23f-4d22-840a-f17d7cd87bb1'(<br />>>>> victor.local.systea.fr) ignoring it in the refresh until migration is<br />>>>> done<br />>>>> ...<br />>>>> 2018-02-12 16:49:31,484+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]<br />>>>> (DefaultQuartzScheduler5) [11a7619a] VM 'f7d4ec12-627a-4b83-b59e-886400d55474'(Oracle_PRIMARY)<br />>>>> was unexpectedly detected as 'MigratingTo' on VDS<br />>>>> 'ce3938b1-b23f-4d22-840a-f17d7cd87bb1'(victor.local.systea.fr)<br />>>>> (expected on 'd569c2dd-8f30-4878-8aea-858db285cf69')<br />>>>> 2018-02-12 16:49:31,484+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]<br />>>>> (DefaultQuartzScheduler5) [11a7619a] VM 'f7d4ec12-627a-4b83-b59e-886400d55474'<br />>>>> is migrating to VDS 'ce3938b1-b23f-4d22-840a-f17d7cd87bb1'(<br />>>>> victor.local.systea.fr) ignoring it in the refresh until migration is<br />>>>> done<br />>>>><br />>>>><br />>>>><br />>>>> and so on, last lines repeated indefinitly for hours since we poweroff<br />>>>> the VM...<br />>>>> Is this something known ? Any idea about that ?<br />>>>><br />>>>> Thanks<br />>>>><br />>>>> Ovirt 4.1.6, updated last at feb-13. Gluster 3.12.1.<br />>>>><br />>>>> --<br />>>>><br />>>>> Cordialement,<br />>>>><br />>>>> *Frank Soyer *<br />>>>><br />>>>> _______________________________________________<br />>>>> Users mailing list<br />>>>> Users@ovirt.org<br />>>>> http://lists.ovirt.org/mailman/listinfo/users<br />>>>><br />>>><br />>>><br />>>><br />>>><br />>><br />>><br />>><br /> </blockquote><br /> </html>