</div></div><div class=3D"gmail=5Fextra"> <div class=3D"gma= il=5Fquote">On Fri, Feb 16, 2018 at 11:03 AM, fsoyer <span dir=3D"ltr">= <<a target=3D"=5Fblank" href=3D"mailto:fsoyer@systea.fr">fsoyer@syst= ea.fr</a>></span> wrote:<blockquote class=3D"gmail=5Fquote" style=3D= "margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Maor= ,<br />sorry for the double post, I've change the email adress of my ac= count and supposed that I'd need to re-post it.<br />And thank you for = your time. Here are the logs. I added a vdisk to an existing VM : it no= more migrates, needing to poweroff it after minutes. Then simply delet= ing the second disk makes migrate it in exactly 9s without problem !&nb= sp;<br /><a target=3D"=5Fblank" href=3D"https://gist.github.com/fgth/47= 07446331d201eef574ac31b6e89561">https://gist.github.com/fgth/<wbr />470= 7446331d201eef574ac31b6e895<wbr />61</a><br /><a target=3D"=5Fblank" hr= ef=3D"https://gist.github.com/fgth/f8de9c22664aee53722af676bff8719d">ht= tps://gist.github.com/fgth/<wbr />f8de9c22664aee53722af676bff871<wbr />= 9d</a><br /><br />--<p class=3D"m=5F-4299273321983674487Text1">Cordiale= ment,<br /><br /><b>Frank Soyer </b></p><div class=3D"HOEnZb"><div= class=3D"h5">Le Mercredi, F=C3=A9vrier 14, 2018 11:04 CET, Maor Lipchu= k <<a target=3D"=5Fblank" href=3D"mailto:mlipchuk@redhat.com">mlipch= uk@redhat.com</a>> a =C3=A9crit:<br /> <blockquote type=3D"cite= " cite=3D"http://CAJ1JNOcD3ZX6hYG4TJ0-=5FumSgw6-wtJoC=5F=5FHRbarc1io-Y-= 6Jw@mail.gmail.com"><div dir=3D"ltr">Hi Frank,<div> </div><div>I a= lready replied on your last email.</div><div><div>Can you provide the V= DSM logs from the time of the migration failure for both hosts:</div><d= iv> <span style=3D"font-family:monospace"> </span><a style=3D= "font-family:monospace" target=3D"=5Fblank" href=3D"http://ginger.local= .systea.fr/">ginger.local.systea.f</a>r and <a style=3D"font-= family:monospace" target=3D"=5Fblank" href=3D"http://victor.local.syste= a.fr/">v<wbr />ictor.local.systea.fr</a></div><div> </div><div>Tha= nks,</div><div>Maor</div></div></div><div class=3D"gmail=5Fextra"> = ;<div class=3D"gmail=5Fquote">On Wed, Feb 14, 2018 at 11:23 AM, fsoyer = <span dir=3D"ltr"><<a target=3D"=5Fblank" href=3D"mailto:fsoyer@syst= ea.fr">fsoyer@systea.fr</a>></span> wrote:<blockquote class=3D"gmail= =5Fquote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding= -left:1ex"><p>Hi all,<br />I discovered yesterday a problem when migrat= ing VM with more than one vdisk.<br />On our test servers (oVirt4.1, sh= ared storage with Gluster), I created 2 VMs needed for a test, from a t= emplate with a 20G vdisk. On this VMs I added a 100G vdisk (for this te= sts I didn't want to waste time to extend the existing vdisks... But I = lost time finally...). The VMs with the 2 vdisks works well.<br />Now I= saw some updates waiting on the host. I tried to put it in maintenance= ... But it stopped on the two VM. They were marked "migrating", but no = more accessible. Other (small) VMs with only 1 vdisk was migrated witho= ut problem at the same time.<br />I saw that a kvm process for the (big= ) VMs was launched on the source AND destination host, but after tens o= f minutes, the migration and the VMs was always freezed. I tried to can= cel the migration for the VMs : failed. The only way to stop it was to =
3da5725] START, MigrateVDSCommand( MigrateVDSCommandParameters:{r<wbr = />unAsync=3D'true', hostId=3D'ce3938b1-b23f-4d22-840<wbr />a-f17d7cd87b= b1', vmId=3D'3f57e669-5e4c-4d10-85cc-<wbr />d573004a099d', srcHost=3D'1= 92.168.0.6', dstVdsId=3D'd569c2dd-8f30-4878-8<wbr />aea-858db285cf69', = dstHost=3D'<a target=3D"=5Fblank" href=3D"http://192.168.0.5:54321">192= .168.0.5:54321</a>', migrationMethod=3D'ONLINE', tunnelMigration=3D'fal= se', migrationDowntime=3D'0', autoConverge=3D'true', migrateCompressed=3D= 'false', consoleAddress=3D'null', maxBandwidth=3D'500', enableGuestEven= ts=3D'true', maxIncomingMigrations=3D'2', maxOutgoingMigrations=3D'2', = convergenceSchedule=3D'[init=3D[{n<wbr />ame=3DsetDowntime, params=3D[1= 00]}], stalling=3D[{limit=3D1, action=3D{name=3DsetDowntime, params=3D[= 150]}}, {limit=3D2, action=3D{name=3DsetDowntime, params=3D[200]}}, {li= mit=3D3, action=3D{name=3DsetDowntime, params=3D[300]}}, {limit=3D4, ac= tion=3D{name=3DsetDowntime, params=3D[400]}}, {limit=3D6, action=3D{nam= e=3DsetDowntime, params=3D[500]}}, {limit=3D-1, action=3D{name=3Dabort,=
broker.auditloghandling.AuditL<wbr />ogDirector] (org.ovirt.thread.poo= l-6-threa<wbr />d-32) [2f712024-5982-46a8-82c8-fd829<wbr />3da5725] EVE= NT=5FID: VM=5FMIGRATION=5FSTART(62), Correlation ID: 2f712024-5982-46a8= -82c8-fd8293<wbr />da5725, Job ID: 4bd19aa9-cc99-4d02-884e-5a1e85<wbr /= 7a7738, Call Stack: null, Custom ID: null, Custom Event ID: -1, Messag= e: Migration started (VM: Oracle=5FSECONDARY, Source: <a target=3D"=5Fb= lank" href=3D"http://victor.local.systea.fr">victor.local.systea.fr</a>= , Destination: <a target=3D"=5Fblank" href=3D"http://ginger.local.syste= a.fr">ginger.local.systea.fr</a>, User: admin@internal-authz).<br />201= 8-02-12 16:46:31,106+01 INFO [org.ovirt.engine.core.vdsbro<wbr />= ker.vdsbroker.FullListVDSComma<wbr />nd] (DefaultQuartzScheduler9) [54a= 65b66] START, FullListVDSCommand(HostName =3D <a target=3D"=5Fblank" hr= ef=3D"http://victor.local.systea.fr">victor.local.systea.fr</a>, FullLi= stVDSCommandParameters:{<wbr />runAsync=3D'true', hostId=3D'ce3938b1-b2= 3f-4d22-840<wbr />a-f17d7cd87bb1', vmIds=3D'[3f57e669-5e4c-4d10-85c<wbr= />c-d573004a099d]'}), log id: 54b4b435<br />2018-02-12 16:46:31,147+01= INFO [org.ovirt.engine.core.vdsbro<wbr />ker.vdsbroker.FullListV= DSComma<wbr />nd] (DefaultQuartzScheduler9) [54a65b66] FINISH, FullList= VDSCommand, return: [{acpiEnable=3Dtrue, emulatedMachine=3Dpc-i440fx-rh= el<wbr />7.3.0, tabletEnable=3Dtrue, pid=3D1493, guestDiskMapping=3D{0Q= EMU=5FQEMU=5FH<wbr />ARDDISK=5Fd890fa68-fba4-4f49-9=3D{<wbr />name=3D/d= ev/sda}, QEMU=5FDVD-ROM=5FQM00003=3D{name=3D/de<wbr />v/sr0}}, transpar= entHugePages=3Dtrue, timeOffset=3D0, cpuType=3DNehalem, smp=3D2, pauseC= ode=3DNOERR, guestNumaNodes=3D[Ljava.lang.Obj<wbr />ect;@1d9042cd, smar= tcardEnable=3Dfalse, custom=3D{device=5Ffbddd528-7d93-4<wbr />9c6-a286-= 180e021cb274device=5F87<wbr />9c93ab-4df1-435c-af02-565039fc<wbr />c254= =3DVmDevice:{id=3D'VmDeviceId:<wbr />{deviceId=3D'879c93ab-4df1-435c-<w= br />af02-565039fcc254', vmId=3D'3f57e669-5e4c-4d10-85cc-<wbr />d573004= a099d'}', device=3D'unix', type=3D'CHANNEL', bootOrder=3D'0', specParam= s=3D'[]', address=3D'{bus=3D0, controller=3D0, type=3Dvirtio-serial, po= rt=3D1}', managed=3D'false', plugged=3D'true', readOnly=3D'false', devi= ceAlias=3D'channel0', customProperties=3D'[]', snapshotId=3D'null', log= icalName=3D'null', hostDevice=3D'null'}, device=5Ffbddd528-7d93-49c6-a2= 86<wbr />-180e021cb274device=5F879c93ab-<wbr />4df1-435c-af02-565039fcc= 254dev<wbr />ice=5F8945f61a-abbe-4156-8485-a4<wbr />aa6f1908dbdevice=5F= 017b5e59-01c4<wbr />-4aac-bf0c-b5d9557284d6=3DVmDevi<wbr />ce:{id=3D'Vm= DeviceId:{deviceId=3D'<wbr />017b5e59-01c4-4aac-bf0c-<wbr />b5d9557284d= 6', vmId=3D'3f57e669-5e4c-4d10-85cc-<wbr />d573004a099d'}', device=3D't= ablet', type=3D'UNKNOWN', bootOrder=3D'0', specParams=3D'[]', address=3D= '{bus=3D0, type=3Dusb, port=3D1}', managed=3D'false', plugged=3D'true',= readOnly=3D'false', deviceAlias=3D'input0', customProperties=3D'[]', s= napshotId=3D'null', logicalName=3D'null', hostDevice=3D'null'}, device=5F= fbddd528-7d93-49c6-a286<wbr />-180e021cb274=3DVmDevice:{id=3D'<wbr />Vm= DeviceId:{deviceId=3D'fbddd528<wbr />-7d93-49c6-a286-180e021cb274', vmI= d=3D'3f57e669-5e4c-4d10-85cc-<wbr />d573004a099d'}', device=3D'ide', ty=
ring] (DefaultQuartzScheduler9) [54a65b66] Received a lease Device wit= hout an address when processing VM 3f57e669-5e4c-4d10-85cc-d57300<wbr /= 4a099d devices, skipping device: {lease=5Fid=3D3f57e669-5e4c-4d10-8<wb= r />5cc-d573004a099d, sd=5Fid=3D1e51cecc-eb2e-47d0-b185-<wbr />920fdc7a= fa16, deviceId=3D{uuid=3Da09949aa-5642-4<wbr />b6d-94a4-8b0d04257be5}, = offset=3D6291456, device=3Dlease, path=3D/rhev/data-center/mnt/glu<wbr = />sterSD/192.168.0.6:=5FDATA01/1e5<wbr />1cecc-eb2e-47d0-b185-920fdc7af= <wbr />a16/dom=5Fmd/xleases, type=3Dlease}<br />2018-02-12 16:46:31,152= +01 INFO [org.ovirt.engine.core.vdsbro<wbr />ker.monitoring.VmAna= lyzer] (DefaultQuartzScheduler1) [27fac647] VM '3f57e669-5e4c-4d10-85cc= -d5730<wbr />04a099d'(Oracle=5FSECONDARY) was unexpectedly detected as = 'MigratingTo' on VDS 'd569c2dd-8f30-4878-8aea-858db<wbr />285cf69'(<a t= arget=3D"=5Fblank" href=3D"http://ginger.local.systea.fr">ginger.local.= systea.<wbr />fr</a>) (expected on 'ce3938b1-b23f-4d22-840a-f17d7<wbr /= cd87bb1')<br />2018-02-12 16:46:31,152+01 INFO [org.ovirt.engine= .core.vdsbro<wbr />ker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1)= [27fac647] VM '3f57e669-5e4c-4d10-85cc-d5730<wbr />04a099d' is migrati= ng to VDS 'd569c2dd-8f30-4878-8aea-858db<wbr />285cf69'(<a target=3D"=5F= blank" href=3D"http://ginger.local.systea.fr">ginger.local.systea.<wbr = />fr</a>) ignoring it in the refresh until migration is done<br />....<= br />2018-02-12 16:46:41,631+01 INFO [org.ovirt.engine.core.vdsbr= o<wbr />ker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] VM '3f= 57e669-5e4c-4d10-85cc-d5730<wbr />04a099d' was reported as Down on VDS = 'ce3938b1-b23f-4d22-840a-f17d7<wbr />cd87bb1'(<a target=3D"=5Fblank" hr= ef=3D"http://victor.local.systea.fr">victor.local.systea.<wbr />fr</a>)= <br />2018-02-12 16:46:41,632+01 INFO [org.ovirt.engine.core.vdsb= ro<wbr />ker.vdsbroker.DestroyVDSComman<wbr />d] (ForkJoinPool-1-worker= -11) [] START, DestroyVDSCommand(HostName =3D <a target=3D"=5Fblank" hr= ef=3D"http://victor.local.systea.fr">victor.local.systea.fr</a>, Destro= yVmVDSCommandParameters:<wbr />{runAsync=3D'true', hostId=3D'ce3938b1-b= 23f-4d22-840<wbr />a-f17d7cd87bb1', vmId=3D'3f57e669-5e4c-4d10-85cc-<wb= r />d573004a099d', force=3D'false', secondsToWait=3D'0', gracefully=3D'= false', reason=3D'', ignoreNoVm=3D'true'}), log id: 560eca57<br />2018-= 02-12 16:46:41,650+01 INFO [org.ovirt.engine.core.vdsbro<wbr />ke= r.vdsbroker.DestroyVDSComman<wbr />d] (ForkJoinPool-1-worker-11) [] FIN= ISH, DestroyVDSCommand, log id: 560eca57<br />2018-02-12 16:46:41,650+0= 1 INFO [org.ovirt.engine.core.vdsbro<wbr />ker.monitoring.VmAnaly= zer] (ForkJoinPool-1-worker-11) [] VM '3f57e669-5e4c-4d10-85cc-d5730<wb= r />04a099d'(Oracle=5FSECONDARY) moved from 'MigratingFrom' --> 'Dow= n'<br />2018-02-12 16:46:41,651+01 INFO [org.ovirt.engine.core.vd= sbro<wbr />ker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] Han= ding over VM '3f57e669-5e4c-4d10-85cc-d5730<wbr />04a099d'(Oracle=5FSEC= ONDARY) to Host 'd569c2dd-8f30-4878-8aea-858db<wbr />285cf69'. Setting = VM to status 'MigratingTo'<br />2018-02-12 16:46:42,163+01 INFO [= org.ovirt.engine.core.vdsbro<wbr />ker.monitoring.VmAnalyzer] (ForkJoin= Pool-1-worker-4) [] VM '3f57e669-5e4c-4d10-85cc-d5730<wbr />04a099d'(Or= acle=5FSECONDARY) moved from 'MigratingTo' --> 'Up'<br />2018-02-12 = 16:46:42,169+01 INFO [org.ovirt.engine.core.vdsbro<wbr />ker.vdsb= roker.MigrateStatusVDS<wbr />Command] (ForkJoinPool-1-worker-4) [] STAR= T, MigrateStatusVDSCommand(HostNa<wbr />me =3D <a target=3D"=5Fblank" h= ref=3D"http://ginger.local.systea.fr">ginger.local.systea.fr</a>, Migra= teStatusVDSCommandParamet<wbr />ers:{runAsync=3D'true', hostId=3D'd569c= 2dd-8f30-4878-8ae<wbr />a-858db285cf69', vmId=3D'3f57e669-5e4c-4d10-85c= c-<wbr />d573004a099d'}), log id: 7a25c281<br />2018-02-12 16:46:42,174= +01 INFO [org.ovirt.engine.core.vdsbro<wbr />ker.vdsbroker.Migrat= eStatusVDS<wbr />Command] (ForkJoinPool-1-worker-4) [] FINISH, MigrateS= tatusVDSCommand, log id: 7a25c281<br />2018-02-12 16:46:42,194+01 INFO = [org.ovirt.engine.core.dal.db<wbr />broker.auditloghandling.Audit= L<wbr />ogDirector] (ForkJoinPool-1-worker-4) [] EVENT=5FID: VM=5FMIGRA= TION=5FDONE(63), Correlation ID: 2f712024-5982-46a8-82c8-fd8293<wbr />d= a5725, Job ID: 4bd19aa9-cc99-4d02-884e-5a1e85<wbr />7a7738, Call Stack:= null, Custom ID: null, Custom Event ID: -1, Message: Migration complet= ed (VM: Oracle=5FSECONDARY, Source: <a target=3D"=5Fblank" href=3D"http= ://victor.local.systea.fr">victor.local.systea.fr</a>, Destination: <a = target=3D"=5Fblank" href=3D"http://ginger.local.systea.fr">ginger.local= .systea.fr</a>, Duration: 11 seconds, Total: 11 seconds, Actual downtim= e: (N/A))<br />2018-02-12 16:46:42,201+01 INFO [org.ovirt.engine.= core.bll.Mi<wbr />grateVmToServerCommand] (ForkJoinPool-1-worker-4) [] = Lock freed to object 'EngineLock:{exclusiveLocks=3D'[<wbr />3f57e669-5e= 4c-4d10-85cc-d57300<wbr />4a099d=3DVM]', sharedLocks=3D''}'<br />2018-0= 2-12 16:46:42,203+01 INFO [org.ovirt.engine.core.vdsbro<wbr />ker= .vdsbroker.FullListVDSComma<wbr />nd] (ForkJoinPool-1-worker-4) [] STAR= T, FullListVDSCommand(HostName =3D <a target=3D"=5Fblank" href=3D"http:= //ginger.local.systea.fr">ginger.local.systea.fr</a>, FullListVDSComman= dParameters:{<wbr />runAsync=3D'true', hostId=3D'd569c2dd-8f30-4878-8ae= <wbr />a-858db285cf69', vmIds=3D'[3f57e669-5e4c-4d10-85c<wbr />c-d57300= 4a099d]'}), log id: 7cc65298<br />2018-02-12 16:46:42,254+01 INFO = ;[org.ovirt.engine.core.vdsbro<wbr />ker.vdsbroker.FullListVDSComma<wbr= />nd] (ForkJoinPool-1-worker-4) [] FINISH, FullListVDSCommand, return:= [{acpiEnable=3Dtrue, emulatedMachine=3Dpc-i440fx-rhel<wbr />7.3.0, aft= erMigrationStatus=3D, tabletEnable=3Dtrue, pid=3D18748, guestDiskMappin= g=3D{}, transparentHugePages=3Dtrue, timeOffset=3D0, cpuType=3DNehalem,= smp=3D2, guestNumaNodes=3D[Ljava.lang.Obj<wbr />ect;@760085fd, custom=3D= {device=5Ffbddd528-7d93-4<wbr />9c6-a286-180e021cb274device=5F87<wbr />= 9c93ab-4df1-435c-af02-565039fc<wbr />c254=3DVmDevice:{id=3D'VmDeviceId:= <wbr />{deviceId=3D'879c93ab-4df1-435c-<wbr />af02-565039fcc254', vmId=3D= '3f57e669-5e4c-4d10-85cc-<wbr />d573004a099d'}', device=3D'unix', type=3D= 'CHANNEL', bootOrder=3D'0', specParams=3D'[]', address=3D'{bus=3D0, con=
d573004a099d'}', device=3D'unix', type=3D'CHANNEL', bootOrder=3D'0', s=
------=_=-_OpenGroupware_org_NGMime-18019-1519309377.654147-22------ Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Length: 31863 Hi, Yes, on 2018-02-16 (vdsm logs) I tried with a VM standing on ginger (19= 2.168.0.6) migrated=C2=A0(or failed to migrate...) to victor (192.168.0= .5), while the engine.log in the first mail on 2018-02-12 was for VMs s= tanding on victor, migrated (or failed to migrate...) to ginger. Sympto= ms were exactly the same, in both directions, and VMs works like a char= m before, and even after (migration "killed" by a poweroff of VMs). Am I the only one experimenting this problem ? Thanks -- Cordialement, Frank Soyer =C2=A0 Le Jeudi, F=C3=A9vrier 22, 2018 00:45 CET, Maor Lipchuk <mlipchuk@redha= t.com> a =C3=A9crit: =C2=A0Hi Frank,=C2=A0Sorry about the delay repond.I've been going throu= gh the logs you attached, although I could not find any specific indica= tion why the migration failed because of the disk you were mentionning.= Does this VM run with both disks on the target host without migration?=C2= =A0Regards,Maor=C2=A0=C2=A0On Fri, Feb 16, 2018 at 11:03 AM, fsoyer <fs= oyer@systea.fr> wrote:Hi Maor, sorry for the double post, I've change the email adress of my account a= nd supposed that I'd need to re-post it. And thank you for your time. Here are the logs. I added a vdisk to an e= xisting VM : it no more migrates, needing to poweroff it after minutes.= Then simply deleting the second disk makes migrate it in exactly 9s wi= thout problem !=C2=A0 https://gist.github.com/fgth/4707446331d201eef574ac31b6e89561 https://gist.github.com/fgth/f8de9c22664aee53722af676bff8719d -- Cordialement, Frank Soyer=C2=A0Le Mercredi, F=C3=A9vrier 14, 2018 11:04 CET, Maor Lip= chuk <mlipchuk@redhat.com> a =C3=A9crit: =C2=A0Hi Frank,=C2=A0I already replied on your last email.Can you provi= de the VDSM logs from the time of the migration failure for both hosts:= =C2=A0=C2=A0ginger.local.systea.fr=C2=A0and=C2=A0victor.local.systea.fr= =C2=A0Thanks,Maor=C2=A0On Wed, Feb 14, 2018 at 11:23 AM, fsoyer <fsoyer= @systea.fr> wrote: Hi all, I discovered yesterday a problem when migrating VM with more than one v= disk. On our test servers (oVirt4.1, shared storage with Gluster), I created = 2 VMs needed for a test, from a template with a 20G vdisk. On this VMs = I added a 100G vdisk (for this tests I didn't want to waste time to ext= end the existing vdisks... But I lost time finally...). The VMs with th= e 2 vdisks works well. Now I saw some updates waiting on the host. I tried to put it in mainte= nance... But it stopped on the two VM. They were marked "migrating", bu= t no more accessible. Other (small) VMs with only 1 vdisk was migrated = without problem at the same time. I saw that a kvm process for the (big) VMs was launched on the source A= ND destination host, but after tens of minutes, the migration and the V= Ms was always freezed. I tried to cancel the migration for the VMs : fa= iled. The only way to stop it was to poweroff the VMs : the kvm process= died on the 2 hosts and the GUI alerted on a failed migration. In doubt, I tried to delete the second vdisk on one of this VMs : it mi= grates then without error ! And no access problem. I tried to extend the first vdisk of the second VM, the delete the seco= nd vdisk : it migrates now without problem !=C2=A0=C2=A0=C2=A0 So after another test with a VM with 2 vdisks, I can say that this bloc= ked the migration process :( In engine.log, for a VMs with 1 vdisk migrating well, we see :2018-02-1= 2 16:46:29,705+01 INFO =C2=A0[org.ovirt.engine.core.bll.MigrateVmToServ= erCommand] (default task-28) [2f712024-5982-46a8-82c8-fd8293da5725] Loc= k Acquired to object 'EngineLock:{exclusiveLocks=3D'[3f57e669-5e4c-4d10= -85cc-d573004a099d=3DVM]', sharedLocks=3D''}' 2018-02-12 16:46:29,955+01 INFO =C2=A0[org.ovirt.engine.core.bll.Migrat= eVmToServerCommand] (org.ovirt.thread.pool-6-thread-32) [2f712024-5982-= 46a8-82c8-fd8293da5725] Running command: MigrateVmToServerCommand inter= nal: false. Entities affected : =C2=A0ID: 3f57e669-5e4c-4d10-85cc-d5730= 04a099d Type: VMAction group MIGRATE=5FVM with role type USER 2018-02-12 16:46:30,261+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2f712024-5982-4= 6a8-82c8-fd8293da5725] START, MigrateVDSCommand( MigrateVDSCommandParam= eters:{runAsync=3D'true', hostId=3D'ce3938b1-b23f-4d22-840a-f17d7cd87bb= 1', vmId=3D'3f57e669-5e4c-4d10-85cc-d573004a099d', srcHost=3D'192.168.0= .6', dstVdsId=3D'd569c2dd-8f30-4878-8aea-858db285cf69', dstHost=3D'192.= 168.0.5:54321', migrationMethod=3D'ONLINE', tunnelMigration=3D'false', = migrationDowntime=3D'0', autoConverge=3D'true', migrateCompressed=3D'fa= lse', consoleAddress=3D'null', maxBandwidth=3D'500', enableGuestEvents=3D= 'true', maxIncomingMigrations=3D'2', maxOutgoingMigrations=3D'2', conve= rgenceSchedule=3D'[init=3D[{name=3DsetDowntime, params=3D[100]}], stall= ing=3D[{limit=3D1, action=3D{name=3DsetDowntime, params=3D[150]}}, {lim= it=3D2, action=3D{name=3DsetDowntime, params=3D[200]}}, {limit=3D3, act= ion=3D{name=3DsetDowntime, params=3D[300]}}, {limit=3D4, action=3D{name= =3DsetDowntime, params=3D[400]}}, {limit=3D6, action=3D{name=3DsetDownt= ime, params=3D[500]}}, {limit=3D-1, action=3D{name=3Dabort, params=3D[]= }}]]'}), log id: 14f61ee0 2018-02-12 16:46:30,262+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-32) = [2f712024-5982-46a8-82c8-fd8293da5725] START, MigrateBrokerVDSCommand(H= ostName =3D victor.local.systea.fr, MigrateVDSCommandParameters:{runAsy= nc=3D'true', hostId=3D'ce3938b1-b23f-4d22-840a-f17d7cd87bb1', vmId=3D'3= f57e669-5e4c-4d10-85cc-d573004a099d', srcHost=3D'192.168.0.6', dstVdsId= =3D'd569c2dd-8f30-4878-8aea-858db285cf69', dstHost=3D'192.168.0.5:54321= ', migrationMethod=3D'ONLINE', tunnelMigration=3D'false', migrationDown= time=3D'0', autoConverge=3D'true', migrateCompressed=3D'false', console= Address=3D'null', maxBandwidth=3D'500', enableGuestEvents=3D'true', max= IncomingMigrations=3D'2', maxOutgoingMigrations=3D'2', convergenceSched= ule=3D'[init=3D[{name=3DsetDowntime, params=3D[100]}], stalling=3D[{lim= it=3D1, action=3D{name=3DsetDowntime, params=3D[150]}}, {limit=3D2, act= ion=3D{name=3DsetDowntime, params=3D[200]}}, {limit=3D3, action=3D{name= =3DsetDowntime, params=3D[300]}}, {limit=3D4, action=3D{name=3DsetDownt= ime, params=3D[400]}}, {limit=3D6, action=3D{name=3DsetDowntime, params= =3D[500]}}, {limit=3D-1, action=3D{name=3Dabort, params=3D[]}}]]'}), lo= g id: 775cd381 2018-02-12 16:46:30,277+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-32) = [2f712024-5982-46a8-82c8-fd8293da5725] FINISH, MigrateBrokerVDSCommand,= log id: 775cd381 2018-02-12 16:46:30,285+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2f712024-5982-4= 6a8-82c8-fd8293da5725] FINISH, MigrateVDSCommand, return: MigratingFrom= , log id: 14f61ee0 2018-02-12 16:46:30,301+01 INFO =C2=A0[org.ovirt.engine.core.dal.dbbrok= er.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3= 2) [2f712024-5982-46a8-82c8-fd8293da5725] EVENT=5FID: VM=5FMIGRATION=5F= START(62), Correlation ID: 2f712024-5982-46a8-82c8-fd8293da5725, Job ID= : 4bd19aa9-cc99-4d02-884e-5a1e857a7738, Call Stack: null, Custom ID: nu= ll, Custom Event ID: -1, Message: Migration started (VM: Oracle=5FSECON= DARY, Source: victor.local.systea.fr, Destination: ginger.local.systea.= fr, User: admin@internal-authz). 2018-02-12 16:46:31,106+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler9) [54a65b66] STAR= T, FullListVDSCommand(HostName =3D victor.local.systea.fr, FullListVDSC= ommandParameters:{runAsync=3D'true', hostId=3D'ce3938b1-b23f-4d22-840a-= f17d7cd87bb1', vmIds=3D'[3f57e669-5e4c-4d10-85cc-d573004a099d]'}), log = id: 54b4b435 2018-02-12 16:46:31,147+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler9) [54a65b66] FINI= SH, FullListVDSCommand, return: [{acpiEnable=3Dtrue, emulatedMachine=3D= pc-i440fx-rhel7.3.0, tabletEnable=3Dtrue, pid=3D1493, guestDiskMapping=3D= {0QEMU=5FQEMU=5FHARDDISK=5Fd890fa68-fba4-4f49-9=3D{name=3D/dev/sda}, QE= MU=5FDVD-ROM=5FQM00003=3D{name=3D/dev/sr0}}, transparentHugePages=3Dtru= e, timeOffset=3D0, cpuType=3DNehalem, smp=3D2, pauseCode=3DNOERR, guest= NumaNodes=3D[Ljava.lang.Object;@1d9042cd, smartcardEnable=3Dfalse, cust= om=3D{device=5Ffbddd528-7d93-49c6-a286-180e021cb274device=5F879c93ab-4d= f1-435c-af02-565039fcc254=3DVmDevice:{id=3D'VmDeviceId:{deviceId=3D'879= c93ab-4df1-435c-af02-565039fcc254', vmId=3D'3f57e669-5e4c-4d10-85cc-d57= 3004a099d'}', device=3D'unix', type=3D'CHANNEL', bootOrder=3D'0', specP= arams=3D'[]', address=3D'{bus=3D0, controller=3D0, type=3Dvirtio-serial= , port=3D1}', managed=3D'false', plugged=3D'true', readOnly=3D'false', = deviceAlias=3D'channel0', customProperties=3D'[]', snapshotId=3D'null',= logicalName=3D'null', hostDevice=3D'null'}, device=5Ffbddd528-7d93-49c= 6-a286-180e021cb274device=5F879c93ab-4df1-435c-af02-565039fcc254device=5F= 8945f61a-abbe-4156-8485-a4aa6f1908dbdevice=5F017b5e59-01c4-4aac-bf0c-b5= d9557284d6=3DVmDevice:{id=3D'VmDeviceId:{deviceId=3D'017b5e59-01c4-4aac= -bf0c-b5d9557284d6', vmId=3D'3f57e669-5e4c-4d10-85cc-d573004a099d'}', d= evice=3D'tablet', type=3D'UNKNOWN', bootOrder=3D'0', specParams=3D'[]',= address=3D'{bus=3D0, type=3Dusb, port=3D1}', managed=3D'false', plugge= d=3D'true', readOnly=3D'false', deviceAlias=3D'input0', customPropertie= s=3D'[]', snapshotId=3D'null', logicalName=3D'null', hostDevice=3D'null= '}, device=5Ffbddd528-7d93-49c6-a286-180e021cb274=3DVmDevice:{id=3D'VmD= eviceId:{deviceId=3D'fbddd528-7d93-49c6-a286-180e021cb274', vmId=3D'3f5= 7e669-5e4c-4d10-85cc-d573004a099d'}', device=3D'ide', type=3D'CONTROLLE= R', bootOrder=3D'0', specParams=3D'[]', address=3D'{slot=3D0x01, bus=3D= 0x00, domain=3D0x0000, type=3Dpci, function=3D0x1}', managed=3D'false',= plugged=3D'true', readOnly=3D'false', deviceAlias=3D'ide', customPrope= rties=3D'[]', snapshotId=3D'null', logicalName=3D'null', hostDevice=3D'= null'}, device=5Ffbddd528-7d93-49c6-a286-180e021cb274device=5F879c93ab-= 4df1-435c-af02-565039fcc254device=5F8945f61a-abbe-4156-8485-a4aa6f1908d= b=3DVmDevice:{id=3D'VmDeviceId:{deviceId=3D'8945f61a-abbe-4156-8485-a4a= a6f1908db', vmId=3D'3f57e669-5e4c-4d10-85cc-d573004a099d'}', device=3D'= unix', type=3D'CHANNEL', bootOrder=3D'0', specParams=3D'[]', address=3D= '{bus=3D0, controller=3D0, type=3Dvirtio-serial, port=3D2}', managed=3D= 'false', plugged=3D'true', readOnly=3D'false', deviceAlias=3D'channel1'= , customProperties=3D'[]', snapshotId=3D'null', logicalName=3D'null', h= ostDevice=3D'null'}}, vmType=3Dkvm, memSize=3D8192, smpCoresPerSocket=3D= 1, vmName=3DOracle=5FSECONDARY, nice=3D0, status=3DMigration Source, ma= xMemSize=3D32768, bootMenuEnable=3Dfalse, vmId=3D3f57e669-5e4c-4d10-85c= c-d573004a099d, numOfIoThreads=3D2, smpThreadsPerCore=3D1, memGuarantee= dSize=3D8192, kvmEnable=3Dtrue, pitReinjection=3Dfalse, displayNetwork=3D= ovirtmgmt, devices=3D[Ljava.lang.Object;@28ae66d7, display=3Dvnc, maxVC= pus=3D16, clientIp=3D, statusTime=3D4299484520, maxMemSlots=3D16}], log= id: 54b4b435 2018-02-12 16:46:31,150+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler1) [27fac647] F= etched 3 VMs from VDS 'd569c2dd-8f30-4878-8aea-858db285cf69' 2018-02-12 16:46:31,151+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9) [54a65b66] Re= ceived a vnc Device without an address when processing VM 3f57e669-5e4c= -4d10-85cc-d573004a099d devices, skipping device: {device=3Dvnc, specPa= rams=3D{displayNetwork=3Dovirtmgmt, keyMap=3Dfr, displayIp=3D192.168.0.= 6}, type=3Dgraphics, deviceId=3D813957b1-446a-4e88-9e40-9fe76d2c442d, p= ort=3D5901} 2018-02-12 16:46:31,151+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9) [54a65b66] Re= ceived a lease Device without an address when processing VM 3f57e669-5e= 4c-4d10-85cc-d573004a099d devices, skipping device: {lease=5Fid=3D3f57e= 669-5e4c-4d10-85cc-d573004a099d, sd=5Fid=3D1e51cecc-eb2e-47d0-b185-920f= dc7afa16, deviceId=3D{uuid=3Da09949aa-5642-4b6d-94a4-8b0d04257be5}, off= set=3D6291456, device=3Dlease, path=3D/rhev/data-center/mnt/glusterSD/1= 92.168.0.6:=5FDATA01/1e51cecc-eb2e-47d0-b185-920fdc7afa16/dom=5Fmd/xlea= ses, type=3Dlease} 2018-02-12 16:46:31,152+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [27fac647] VM '3f57e66= 9-5e4c-4d10-85cc-d573004a099d'(Oracle=5FSECONDARY) was unexpectedly det= ected as 'MigratingTo' on VDS 'd569c2dd-8f30-4878-8aea-858db285cf69'(gi= nger.local.systea.fr) (expected on 'ce3938b1-b23f-4d22-840a-f17d7cd87bb= 1') 2018-02-12 16:46:31,152+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [27fac647] VM '3f57e66= 9-5e4c-4d10-85cc-d573004a099d' is migrating to VDS 'd569c2dd-8f30-4878-= 8aea-858db285cf69'(ginger.local.systea.fr) ignoring it in the refresh u= ntil migration is done .... 2018-02-12 16:46:41,631+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] VM '3f57e669-5e4c-= 4d10-85cc-d573004a099d' was reported as Down on VDS 'ce3938b1-b23f-4d22= -840a-f17d7cd87bb1'(victor.local.systea.fr) 2018-02-12 16:46:41,632+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-11) [] START, Destr= oyVDSCommand(HostName =3D victor.local.systea.fr, DestroyVmVDSCommandPa= rameters:{runAsync=3D'true', hostId=3D'ce3938b1-b23f-4d22-840a-f17d7cd8= 7bb1', vmId=3D'3f57e669-5e4c-4d10-85cc-d573004a099d', force=3D'false', = secondsToWait=3D'0', gracefully=3D'false', reason=3D'', ignoreNoVm=3D't= rue'}), log id: 560eca57 2018-02-12 16:46:41,650+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-11) [] FINISH, Dest= royVDSCommand, log id: 560eca57 2018-02-12 16:46:41,650+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] VM '3f57e669-5e4c-= 4d10-85cc-d573004a099d'(Oracle=5FSECONDARY) moved from 'MigratingFrom' = --> 'Down' 2018-02-12 16:46:41,651+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] Handing over VM '3= f57e669-5e4c-4d10-85cc-d573004a099d'(Oracle=5FSECONDARY) to Host 'd569c= 2dd-8f30-4878-8aea-858db285cf69'. Setting VM to status 'MigratingTo' 2018-02-12 16:46:42,163+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmAnalyzer] (ForkJoinPool-1-worker-4) [] VM '3f57e669-5e4c-4= d10-85cc-d573004a099d'(Oracle=5FSECONDARY) moved from 'MigratingTo' -->= 'Up' 2018-02-12 16:46:42,169+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.MigrateStatusVDSCommand] (ForkJoinPool-1-worker-4) [] START, = MigrateStatusVDSCommand(HostName =3D ginger.local.systea.fr, MigrateSta= tusVDSCommandParameters:{runAsync=3D'true', hostId=3D'd569c2dd-8f30-487= 8-8aea-858db285cf69', vmId=3D'3f57e669-5e4c-4d10-85cc-d573004a099d'}), = log id: 7a25c281 2018-02-12 16:46:42,174+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.MigrateStatusVDSCommand] (ForkJoinPool-1-worker-4) [] FINISH,= MigrateStatusVDSCommand, log id: 7a25c281 2018-02-12 16:46:42,194+01 INFO =C2=A0[org.ovirt.engine.core.dal.dbbrok= er.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-4) [] EVEN= T=5FID: VM=5FMIGRATION=5FDONE(63), Correlation ID: 2f712024-5982-46a8-8= 2c8-fd8293da5725, Job ID: 4bd19aa9-cc99-4d02-884e-5a1e857a7738, Call St= ack: null, Custom ID: null, Custom Event ID: -1, Message: Migration com= pleted (VM: Oracle=5FSECONDARY, Source: victor.local.systea.fr, Destina= tion: ginger.local.systea.fr, Duration: 11 seconds, Total: 11 seconds, = Actual downtime: (N/A)) 2018-02-12 16:46:42,201+01 INFO =C2=A0[org.ovirt.engine.core.bll.Migrat= eVmToServerCommand] (ForkJoinPool-1-worker-4) [] Lock freed to object '= EngineLock:{exclusiveLocks=3D'[3f57e669-5e4c-4d10-85cc-d573004a099d=3DV= M]', sharedLocks=3D''}' 2018-02-12 16:46:42,203+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-4) [] START, FullL= istVDSCommand(HostName =3D ginger.local.systea.fr, FullListVDSCommandPa= rameters:{runAsync=3D'true', hostId=3D'd569c2dd-8f30-4878-8aea-858db285= cf69', vmIds=3D'[3f57e669-5e4c-4d10-85cc-d573004a099d]'}), log id: 7cc6= 5298 2018-02-12 16:46:42,254+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-4) [] FINISH, Full= ListVDSCommand, return: [{acpiEnable=3Dtrue, emulatedMachine=3Dpc-i440f= x-rhel7.3.0, afterMigrationStatus=3D, tabletEnable=3Dtrue, pid=3D18748,= guestDiskMapping=3D{}, transparentHugePages=3Dtrue, timeOffset=3D0, cp= uType=3DNehalem, smp=3D2, guestNumaNodes=3D[Ljava.lang.Object;@760085fd= , custom=3D{device=5Ffbddd528-7d93-49c6-a286-180e021cb274device=5F879c9= 3ab-4df1-435c-af02-565039fcc254=3DVmDevice:{id=3D'VmDeviceId:{deviceId=3D= '879c93ab-4df1-435c-af02-565039fcc254', vmId=3D'3f57e669-5e4c-4d10-85cc= -d573004a099d'}', device=3D'unix', type=3D'CHANNEL', bootOrder=3D'0', s= pecParams=3D'[]', address=3D'{bus=3D0, controller=3D0, type=3Dvirtio-se= rial, port=3D1}', managed=3D'false', plugged=3D'true', readOnly=3D'fals= e', deviceAlias=3D'channel0', customProperties=3D'[]', snapshotId=3D'nu= ll', logicalName=3D'null', hostDevice=3D'null'}, device=5Ffbddd528-7d93= -49c6-a286-180e021cb274device=5F879c93ab-4df1-435c-af02-565039fcc254dev= ice=5F8945f61a-abbe-4156-8485-a4aa6f1908dbdevice=5F017b5e59-01c4-4aac-b= f0c-b5d9557284d6=3DVmDevice:{id=3D'VmDeviceId:{deviceId=3D'017b5e59-01c= 4-4aac-bf0c-b5d9557284d6', vmId=3D'3f57e669-5e4c-4d10-85cc-d573004a099d= '}', device=3D'tablet', type=3D'UNKNOWN', bootOrder=3D'0', specParams=3D= '[]', address=3D'{bus=3D0, type=3Dusb, port=3D1}', managed=3D'false', p= lugged=3D'true', readOnly=3D'false', deviceAlias=3D'input0', customProp= erties=3D'[]', snapshotId=3D'null', logicalName=3D'null', hostDevice=3D= 'null'}, device=5Ffbddd528-7d93-49c6-a286-180e021cb274=3DVmDevice:{id=3D= 'VmDeviceId:{deviceId=3D'fbddd528-7d93-49c6-a286-180e021cb274', vmId=3D= '3f57e669-5e4c-4d10-85cc-d573004a099d'}', device=3D'ide', type=3D'CONTR= OLLER', bootOrder=3D'0', specParams=3D'[]', address=3D'{slot=3D0x01, bu= s=3D0x00, domain=3D0x0000, type=3Dpci, function=3D0x1}', managed=3D'fal= se', plugged=3D'true', readOnly=3D'false', deviceAlias=3D'ide', customP= roperties=3D'[]', snapshotId=3D'null', logicalName=3D'null', hostDevice= =3D'null'}, device=5Ffbddd528-7d93-49c6-a286-180e021cb274device=5F879c9= 3ab-4df1-435c-af02-565039fcc254device=5F8945f61a-abbe-4156-8485-a4aa6f1= 908db=3DVmDevice:{id=3D'VmDeviceId:{deviceId=3D'8945f61a-abbe-4156-8485= -a4aa6f1908db', vmId=3D'3f57e669-5e4c-4d10-85cc-d573004a099d'}', device= =3D'unix', type=3D'CHANNEL', bootOrder=3D'0', specParams=3D'[]', addres= s=3D'{bus=3D0, controller=3D0, type=3Dvirtio-serial, port=3D2}', manage= d=3D'false', plugged=3D'true', readOnly=3D'false', deviceAlias=3D'chann= el1', customProperties=3D'[]', snapshotId=3D'null', logicalName=3D'null= ', hostDevice=3D'null'}}, vmType=3Dkvm, memSize=3D8192, smpCoresPerSock= et=3D1, vmName=3DOracle=5FSECONDARY, nice=3D0, status=3DUp, maxMemSize=3D= 32768, bootMenuEnable=3Dfalse, vmId=3D3f57e669-5e4c-4d10-85cc-d573004a0= 99d, numOfIoThreads=3D2, smpThreadsPerCore=3D1, smartcardEnable=3Dfalse= , maxMemSlots=3D16, kvmEnable=3Dtrue, pitReinjection=3Dfalse, displayNe= twork=3Dovirtmgmt, devices=3D[Ljava.lang.Object;@2e4d3dd3, memGuarantee= dSize=3D8192, maxVCpus=3D16, clientIp=3D, statusTime=3D4304259600, disp= lay=3Dvnc}], log id: 7cc65298 2018-02-12 16:46:42,257+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-4) [] Received a= vnc Device without an address when processing VM 3f57e669-5e4c-4d10-85= cc-d573004a099d devices, skipping device: {device=3Dvnc, specParams=3D{= displayNetwork=3Dovirtmgmt, keyMap=3Dfr, displayIp=3D192.168.0.5}, type= =3Dgraphics, deviceId=3D813957b1-446a-4e88-9e40-9fe76d2c442d, port=3D59= 01} 2018-02-12 16:46:42,257+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-4) [] Received a= lease Device without an address when processing VM 3f57e669-5e4c-4d10-= 85cc-d573004a099d devices, skipping device: {lease=5Fid=3D3f57e669-5e4c= -4d10-85cc-d573004a099d, sd=5Fid=3D1e51cecc-eb2e-47d0-b185-920fdc7afa16= , deviceId=3D{uuid=3Da09949aa-5642-4b6d-94a4-8b0d04257be5}, offset=3D62= 91456, device=3Dlease, path=3D/rhev/data-center/mnt/glusterSD/192.168.0= .6:=5FDATA01/1e51cecc-eb2e-47d0-b185-920fdc7afa16/dom=5Fmd/xleases, typ= e=3Dlease} 2018-02-12 16:46:46,260+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler5) [7fcb200a] FINI= SH, FullListVDSCommand, return: [{acpiEnable=3Dtrue, emulatedMachine=3D= pc-i440fx-rhel7.3.0, afterMigrationStatus=3D, tabletEnable=3Dtrue, pid=3D= 18748, guestDiskMapping=3D{0QEMU=5FQEMU=5FHARDDISK=5Fd890fa68-fba4-4f49= -9=3D{name=3D/dev/sda}, QEMU=5FDVD-ROM=5FQM00003=3D{name=3D/dev/sr0}}, = transparentHugePages=3Dtrue, timeOffset=3D0, cpuType=3DNehalem, smp=3D2= , guestNumaNodes=3D[Ljava.lang.Object;@77951faf, custom=3D{device=5Ffbd= dd528-7d93-49c6-a286-180e021cb274device=5F879c93ab-4df1-435c-af02-56503= 9fcc254=3DVmDevice:{id=3D'VmDeviceId:{deviceId=3D'879c93ab-4df1-435c-af= 02-565039fcc254', vmId=3D'3f57e669-5e4c-4d10-85cc-d573004a099d'}', devi= ce=3D'unix', type=3D'CHANNEL', bootOrder=3D'0', specParams=3D'[]', addr= ess=3D'{bus=3D0, controller=3D0, type=3Dvirtio-serial, port=3D1}', mana= ged=3D'false', plugged=3D'true', readOnly=3D'false', deviceAlias=3D'cha= nnel0', customProperties=3D'[]', snapshotId=3D'null', logicalName=3D'nu= ll', hostDevice=3D'null'}, device=5Ffbddd528-7d93-49c6-a286-180e021cb27= 4device=5F879c93ab-4df1-435c-af02-565039fcc254device=5F8945f61a-abbe-41= 56-8485-a4aa6f1908dbdevice=5F017b5e59-01c4-4aac-bf0c-b5d9557284d6=3DVmD= evice:{id=3D'VmDeviceId:{deviceId=3D'017b5e59-01c4-4aac-bf0c-b5d9557284= d6', vmId=3D'3f57e669-5e4c-4d10-85cc-d573004a099d'}', device=3D'tablet'= , type=3D'UNKNOWN', bootOrder=3D'0', specParams=3D'[]', address=3D'{bus= =3D0, type=3Dusb, port=3D1}', managed=3D'false', plugged=3D'true', read= Only=3D'false', deviceAlias=3D'input0', customProperties=3D'[]', snapsh= otId=3D'null', logicalName=3D'null', hostDevice=3D'null'}, device=5Ffbd= dd528-7d93-49c6-a286-180e021cb274=3DVmDevice:{id=3D'VmDeviceId:{deviceI= d=3D'fbddd528-7d93-49c6-a286-180e021cb274', vmId=3D'3f57e669-5e4c-4d10-= 85cc-d573004a099d'}', device=3D'ide', type=3D'CONTROLLER', bootOrder=3D= '0', specParams=3D'[]', address=3D'{slot=3D0x01, bus=3D0x00, domain=3D0= x0000, type=3Dpci, function=3D0x1}', managed=3D'false', plugged=3D'true= ', readOnly=3D'false', deviceAlias=3D'ide', customProperties=3D'[]', sn= apshotId=3D'null', logicalName=3D'null', hostDevice=3D'null'}, device=5F= fbddd528-7d93-49c6-a286-180e021cb274device=5F879c93ab-4df1-435c-af02-56= 5039fcc254device=5F8945f61a-abbe-4156-8485-a4aa6f1908db=3DVmDevice:{id=3D= 'VmDeviceId:{deviceId=3D'8945f61a-abbe-4156-8485-a4aa6f1908db', vmId=3D= '3f57e669-5e4c-4d10-85cc-d573004a099d'}', device=3D'unix', type=3D'CHAN= NEL', bootOrder=3D'0', specParams=3D'[]', address=3D'{bus=3D0, controll= er=3D0, type=3Dvirtio-serial, port=3D2}', managed=3D'false', plugged=3D= 'true', readOnly=3D'false', deviceAlias=3D'channel1', customProperties=3D= '[]', snapshotId=3D'null', logicalName=3D'null', hostDevice=3D'null'}},= vmType=3Dkvm, memSize=3D8192, smpCoresPerSocket=3D1, vmName=3DOracle=5F= SECONDARY, nice=3D0, status=3DUp, maxMemSize=3D32768, bootMenuEnable=3D= false, vmId=3D3f57e669-5e4c-4d10-85cc-d573004a099d, numOfIoThreads=3D2,= smpThreadsPerCore=3D1, smartcardEnable=3Dfalse, maxMemSlots=3D16, kvmE= nable=3Dtrue, pitReinjection=3Dfalse, displayNetwork=3Dovirtmgmt, devic= es=3D[Ljava.lang.Object;@286410fd, memGuaranteedSize=3D8192, maxVCpus=3D= 16, clientIp=3D, statusTime=3D4304263620, display=3Dvnc}], log id: 58cd= ef4c 2018-02-12 16:46:46,267+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [7fcb200a] Re= ceived a vnc Device without an address when processing VM 3f57e669-5e4c= -4d10-85cc-d573004a099d devices, skipping device: {device=3Dvnc, specPa= rams=3D{displayNetwork=3Dovirtmgmt, keyMap=3Dfr, displayIp=3D192.168.0.= 5}, type=3Dgraphics, deviceId=3D813957b1-446a-4e88-9e40-9fe76d2c442d, p= ort=3D5901} 2018-02-12 16:46:46,268+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [7fcb200a] Re= ceived a lease Device without an address when processing VM 3f57e669-5e= 4c-4d10-85cc-d573004a099d devices, skipping device: {lease=5Fid=3D3f57e= 669-5e4c-4d10-85cc-d573004a099d, sd=5Fid=3D1e51cecc-eb2e-47d0-b185-920f= dc7afa16, deviceId=3D{uuid=3Da09949aa-5642-4b6d-94a4-8b0d04257be5}, off= set=3D6291456, device=3Dlease, path=3D/rhev/data-center/mnt/glusterSD/1= 92.168.0.6:=5FDATA01/1e51cecc-eb2e-47d0-b185-920fdc7afa16/dom=5Fmd/xlea= ses, type=3Dlease} =C2=A0 For the VM with 2 vdisks we see : 2018-02-12 16:49:06,112+01 INFO =C2=A0[org.ovirt.engine.core.bll.Migrat= eVmToServerCommand] (default task-50) [92b5af33-cb87-4142-b8fe-8b838dd7= 458e] Lock Acquired to object 'EngineLock:{exclusiveLocks=3D'[f7d4ec12-= 627a-4b83-b59e-886400d55474=3DVM]', sharedLocks=3D''}' 2018-02-12 16:49:06,407+01 INFO =C2=A0[org.ovirt.engine.core.bll.Migrat= eVmToServerCommand] (org.ovirt.thread.pool-6-thread-49) [92b5af33-cb87-= 4142-b8fe-8b838dd7458e] Running command: MigrateVmToServerCommand inter= nal: false. Entities affected : =C2=A0ID: f7d4ec12-627a-4b83-b59e-88640= 0d55474 Type: VMAction group MIGRATE=5FVM with role type USER 2018-02-12 16:49:06,712+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-49) [92b5af33-cb87-4= 142-b8fe-8b838dd7458e] START, MigrateVDSCommand( MigrateVDSCommandParam= eters:{runAsync=3D'true', hostId=3D'd569c2dd-8f30-4878-8aea-858db285cf6= 9', vmId=3D'f7d4ec12-627a-4b83-b59e-886400d55474', srcHost=3D'192.168.0= .5', dstVdsId=3D'ce3938b1-b23f-4d22-840a-f17d7cd87bb1', dstHost=3D'192.= 168.0.6:54321', migrationMethod=3D'ONLINE', tunnelMigration=3D'false', = migrationDowntime=3D'0', autoConverge=3D'true', migrateCompressed=3D'fa= lse', consoleAddress=3D'null', maxBandwidth=3D'500', enableGuestEvents=3D= 'true', maxIncomingMigrations=3D'2', maxOutgoingMigrations=3D'2', conve= rgenceSchedule=3D'[init=3D[{name=3DsetDowntime, params=3D[100]}], stall= ing=3D[{limit=3D1, action=3D{name=3DsetDowntime, params=3D[150]}}, {lim= it=3D2, action=3D{name=3DsetDowntime, params=3D[200]}}, {limit=3D3, act= ion=3D{name=3DsetDowntime, params=3D[300]}}, {limit=3D4, action=3D{name= =3DsetDowntime, params=3D[400]}}, {limit=3D6, action=3D{name=3DsetDownt= ime, params=3D[500]}}, {limit=3D-1, action=3D{name=3Dabort, params=3D[]= }}]]'}), log id: 3702a9e0 2018-02-12 16:49:06,713+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-49) = [92b5af33-cb87-4142-b8fe-8b838dd7458e] START, MigrateBrokerVDSCommand(H= ostName =3D ginger.local.systea.fr, MigrateVDSCommandParameters:{runAsy= nc=3D'true', hostId=3D'd569c2dd-8f30-4878-8aea-858db285cf69', vmId=3D'f= 7d4ec12-627a-4b83-b59e-886400d55474', srcHost=3D'192.168.0.5', dstVdsId= =3D'ce3938b1-b23f-4d22-840a-f17d7cd87bb1', dstHost=3D'192.168.0.6:54321= ', migrationMethod=3D'ONLINE', tunnelMigration=3D'false', migrationDown= time=3D'0', autoConverge=3D'true', migrateCompressed=3D'false', console= Address=3D'null', maxBandwidth=3D'500', enableGuestEvents=3D'true', max= IncomingMigrations=3D'2', maxOutgoingMigrations=3D'2', convergenceSched= ule=3D'[init=3D[{name=3DsetDowntime, params=3D[100]}], stalling=3D[{lim= it=3D1, action=3D{name=3DsetDowntime, params=3D[150]}}, {limit=3D2, act= ion=3D{name=3DsetDowntime, params=3D[200]}}, {limit=3D3, action=3D{name= =3DsetDowntime, params=3D[300]}}, {limit=3D4, action=3D{name=3DsetDownt= ime, params=3D[400]}}, {limit=3D6, action=3D{name=3DsetDowntime, params= =3D[500]}}, {limit=3D-1, action=3D{name=3Dabort, params=3D[]}}]]'}), lo= g id: 1840069c 2018-02-12 16:49:06,724+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-49) = [92b5af33-cb87-4142-b8fe-8b838dd7458e] FINISH, MigrateBrokerVDSCommand,= log id: 1840069c 2018-02-12 16:49:06,732+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-49) [92b5af33-cb87-4= 142-b8fe-8b838dd7458e] FINISH, MigrateVDSCommand, return: MigratingFrom= , log id: 3702a9e0 2018-02-12 16:49:06,753+01 INFO =C2=A0[org.ovirt.engine.core.dal.dbbrok= er.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4= 9) [92b5af33-cb87-4142-b8fe-8b838dd7458e] EVENT=5FID: VM=5FMIGRATION=5F= START(62), Correlation ID: 92b5af33-cb87-4142-b8fe-8b838dd7458e, Job ID= : f4f54054-f7c8-4481-8eda-d5a15c383061, Call Stack: null, Custom ID: nu= ll, Custom Event ID: -1, Message: Migration started (VM: Oracle=5FPRIMA= RY, Source: ginger.local.systea.fr, Destination: victor.local.systea.fr= , User: admin@internal-authz). ... 2018-02-12 16:49:16,453+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler4) [162a5bc3] F= etched 2 VMs from VDS 'ce3938b1-b23f-4d22-840a-f17d7cd87bb1' 2018-02-12 16:49:16,455+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [162a5bc3] VM 'f7d4ec1= 2-627a-4b83-b59e-886400d55474'(Oracle=5FPRIMARY) was unexpectedly detec= ted as 'MigratingTo' on VDS 'ce3938b1-b23f-4d22-840a-f17d7cd87bb1'(vict= or.local.systea.fr) (expected on 'd569c2dd-8f30-4878-8aea-858db285cf69'= ) 2018-02-12 16:49:16,455+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [162a5bc3] VM 'f7d4ec1= 2-627a-4b83-b59e-886400d55474' is migrating to VDS 'ce3938b1-b23f-4d22-= 840a-f17d7cd87bb1'(victor.local.systea.fr) ignoring it in the refresh u= ntil migration is done ... 2018-02-12 16:49:31,484+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [11a7619a] VM 'f7d4ec1= 2-627a-4b83-b59e-886400d55474'(Oracle=5FPRIMARY) was unexpectedly detec= ted as 'MigratingTo' on VDS 'ce3938b1-b23f-4d22-840a-f17d7cd87bb1'(vict= or.local.systea.fr) (expected on 'd569c2dd-8f30-4878-8aea-858db285cf69'= ) 2018-02-12 16:49:31,484+01 INFO =C2=A0[org.ovirt.engine.core.vdsbroker.= monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [11a7619a] VM 'f7d4ec1= 2-627a-4b83-b59e-886400d55474' is migrating to VDS 'ce3938b1-b23f-4d22-= 840a-f17d7cd87bb1'(victor.local.systea.fr) ignoring it in the refresh u= ntil migration is done =C2=A0 and so on, last lines repeated indefinitly for hours since we poweroff = the VM... Is this something known ? Any idea about that ? Thanks Ovirt 4.1.6, updated last at feb-13. Gluster 3.12.1. -- Cordialement, Frank Soyer =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users =C2=A0 =C2=A0 =C2=A0 ------=_=-_OpenGroupware_org_NGMime-18019-1519309377.654147-22------ Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Length: 38724 <html>Hi,<br />Yes, on 2018-02-16 (vdsm logs) I tried with a VM standin= g on ginger (192.168.0.6) migrated (or failed to migrate...) to vi= ctor (192.168.0.5), while the engine.log in the first mail on 2018-02-1= 2 was for VMs standing on victor, migrated (or failed to migrate...) to= ginger. Symptoms were exactly the same, in both directions, and VMs wo= rks like a charm before, and even after (migration "killed" by a powero= ff of VMs).<br />Am I the only one experimenting this problem ?<br /><b= r /><br />Thanks<br />--<br /><style type=3D"text/css">.Text1 { color: black; font-size:9pt; font-family:Verdana; } .Text2 { color: black; font-size:7pt; font-family:Verdana; }</style><p class=3D"Text1">Cordialement,<br /><br /><b>Frank Soyer= </b><br /> </p><br /><br />Le Jeudi, F=C3=A9vrier 22, 2018 00:45 = CET, Maor Lipchuk <mlipchuk@redhat.com> a =C3=A9crit:<br /> = <blockquote type=3D"cite" cite=3D"CAJ1JNOe9Yi5XnFWvqOYhpoMuhkXOKAR=3DNO= WafRkRHLXuOTtwtg@mail.gmail.com"><div dir=3D"ltr">Hi Frank,<div> <= /div><div>Sorry about the delay repond.</div><div>I've been going throu= gh the logs you attached, although I could not find any specific indica= tion why the migration failed because of the disk you were mentionning.= </div><div>Does this VM run with both disks on the target host without = migration?</div><div> </div><div>Regards,</div><div>Maor</div><div= poweroff the VMs : the kvm process died on the 2 hosts and the GUI aler= ted on a failed migration.<br />In doubt, I tried to delete the second = vdisk on one of this VMs : it migrates then without error ! And no acce= ss problem.<br />I tried to extend the first vdisk of the second VM, th= e delete the second vdisk : it migrates now without problem ! &nbs= p; <br /><br />So after another test with a VM with 2 vdisks, I ca= n say that this blocked the migration process :(<br /><br />In engine.l= og, for a VMs with 1 vdisk migrating well, we see :</p><blockquote>2018= -02-12 16:46:29,705+01 INFO [org.ovirt.engine.core.bll.Mi<wbr />g= rateVmToServerCommand] (default task-28) [2f712024-5982-46a8-82c8-fd829= <wbr />3da5725] Lock Acquired to object 'EngineLock:{exclusiveLocks=3D'= [<wbr />3f57e669-5e4c-4d10-85cc-d57300<wbr />4a099d=3DVM]', sharedLocks= =3D''}'<br />2018-02-12 16:46:29,955+01 INFO [org.ovirt.engine.co= re.bll.Mi<wbr />grateVmToServerCommand] (org.ovirt.thread.pool-6-threa<= wbr />d-32) [2f712024-5982-46a8-82c8-fd829<wbr />3da5725] Running comma= nd: MigrateVmToServerCommand internal: false. Entities affected : = ;ID: 3f57e669-5e4c-4d10-85cc-d57300<wbr />4a099d Type: VMAction group M= IGRATE=5FVM with role type USER<br />2018-02-12 16:46:30,261+01 INFO &n= bsp;[org.ovirt.engine.core.vdsbro<wbr />ker.MigrateVDSCommand] (org.ovi= rt.thread.pool-6-threa<wbr />d-32) [2f712024-5982-46a8-82c8-fd829<wbr /= params=3D[]}}]]'}), log id: 14f61ee0<br />2018-02-12 16:46:30,262+01 I= NFO [org.ovirt.engine.core.vdsbro<wbr />ker.vdsbroker.MigrateBrok= erVDS<wbr />Command] (org.ovirt.thread.pool-6-threa<wbr />d-32) [2f7120= 24-5982-46a8-82c8-fd829<wbr />3da5725] START, MigrateBrokerVDSCommand(H= ostNa<wbr />me =3D <a target=3D"=5Fblank" href=3D"http://victor.local.s= ystea.fr">victor.local.systea.fr</a>, MigrateVDSCommandParameters:{r<wb= r />unAsync=3D'true', hostId=3D'ce3938b1-b23f-4d22-840<wbr />a-f17d7cd8= 7bb1', vmId=3D'3f57e669-5e4c-4d10-85cc-<wbr />d573004a099d', srcHost=3D= '192.168.0.6', dstVdsId=3D'd569c2dd-8f30-4878-8<wbr />aea-858db285cf69'= , dstHost=3D'<a target=3D"=5Fblank" href=3D"http://192.168.0.5:54321">1= 92.168.0.5:54321</a>', migrationMethod=3D'ONLINE', tunnelMigration=3D'f= alse', migrationDowntime=3D'0', autoConverge=3D'true', migrateCompresse= d=3D'false', consoleAddress=3D'null', maxBandwidth=3D'500', enableGuest= Events=3D'true', maxIncomingMigrations=3D'2', maxOutgoingMigrations=3D'= 2', convergenceSchedule=3D'[init=3D[{n<wbr />ame=3DsetDowntime, params=3D= [100]}], stalling=3D[{limit=3D1, action=3D{name=3DsetDowntime, params=3D= [150]}}, {limit=3D2, action=3D{name=3DsetDowntime, params=3D[200]}}, {l= imit=3D3, action=3D{name=3DsetDowntime, params=3D[300]}}, {limit=3D4, a= ction=3D{name=3DsetDowntime, params=3D[400]}}, {limit=3D6, action=3D{na= me=3DsetDowntime, params=3D[500]}}, {limit=3D-1, action=3D{name=3Dabort= , params=3D[]}}]]'}), log id: 775cd381<br />2018-02-12 16:46:30,277+01 = INFO [org.ovirt.engine.core.vdsbro<wbr />ker.vdsbroker.MigrateBro= kerVDS<wbr />Command] (org.ovirt.thread.pool-6-threa<wbr />d-32) [2f712= 024-5982-46a8-82c8-fd829<wbr />3da5725] FINISH, MigrateBrokerVDSCommand= , log id: 775cd381<br />2018-02-12 16:46:30,285+01 INFO [org.ovir= t.engine.core.vdsbro<wbr />ker.MigrateVDSCommand] (org.ovirt.thread.poo= l-6-threa<wbr />d-32) [2f712024-5982-46a8-82c8-fd829<wbr />3da5725] FIN= ISH, MigrateVDSCommand, return: MigratingFrom, log id: 14f61ee0<br />20= 18-02-12 16:46:30,301+01 INFO [org.ovirt.engine.core.dal.db<wbr /= pe=3D'CONTROLLER', bootOrder=3D'0', specParams=3D'[]', address=3D'{slot= =3D0x01, bus=3D0x00, domain=3D0x0000, type=3Dpci, function=3D0x1}', man= aged=3D'false', plugged=3D'true', readOnly=3D'false', deviceAlias=3D'id= e', customProperties=3D'[]', snapshotId=3D'null', logicalName=3D'null',= hostDevice=3D'null'}, device=5Ffbddd528-7d93-49c6-a286<wbr />-180e021c= b274device=5F879c93ab-<wbr />4df1-435c-af02-565039fcc254dev<wbr />ice=5F= 8945f61a-abbe-4156-8485-<wbr />a4aa6f1908db=3DVmDevice:{id=3D'<wbr />Vm= DeviceId:{deviceId=3D'<wbr />8945f61a-abbe-4156-8485-<wbr />a4aa6f1908d= b', vmId=3D'3f57e669-5e4c-4d10-85cc-<wbr />d573004a099d'}', device=3D'u= nix', type=3D'CHANNEL', bootOrder=3D'0', specParams=3D'[]', address=3D'= {bus=3D0, controller=3D0, type=3Dvirtio-serial, port=3D2}', managed=3D'= false', plugged=3D'true', readOnly=3D'false', deviceAlias=3D'channel1',= customProperties=3D'[]', snapshotId=3D'null', logicalName=3D'null', ho= stDevice=3D'null'}}, vmType=3Dkvm, memSize=3D8192, smpCoresPerSocket=3D= 1, vmName=3DOracle=5FSECONDARY, nice=3D0, status=3DMigration Source, ma= xMemSize=3D32768, bootMenuEnable=3Dfalse, vmId=3D3f57e669-5e4c-4d10-85c= c-d<wbr />573004a099d, numOfIoThreads=3D2, smpThreadsPerCore=3D1, memGu= aranteedSize=3D8192, kvmEnable=3Dtrue, pitReinjection=3Dfalse, displayN= etwork=3Dovirtmgmt, devices=3D[Ljava.lang.Object;@28<wbr />ae66d7, disp= lay=3Dvnc, maxVCpus=3D16, clientIp=3D, statusTime=3D4299484520, maxMemS= lots=3D16}], log id: 54b4b435<br />2018-02-12 16:46:31,150+01 INFO &nbs= p;[org.ovirt.engine.core.vdsbro<wbr />ker.monitoring.VmsStatisticsFe<wb= r />tcher] (DefaultQuartzScheduler1) [27fac647] Fetched 3 VMs from VDS = 'd569c2dd-8f30-4878-8aea-858db<wbr />285cf69'<br />2018-02-12 16:46:31,= 151+01 INFO [org.ovirt.engine.core.vdsbro<wbr />ker.monitoring.Vm= DevicesMonito<wbr />ring] (DefaultQuartzScheduler9) [54a65b66] Received= a vnc Device without an address when processing VM 3f57e669-5e4c-4d10-= 85cc-d57300<wbr />4a099d devices, skipping device: {device=3Dvnc, specP= arams=3D{displayNetwork=3Dovi<wbr />rtmgmt, keyMap=3Dfr, displayIp=3D19= 2.168.0.6}, type=3Dgraphics, deviceId=3D813957b1-446a-4e88-9e<wbr />40-= 9fe76d2c442d, port=3D5901}<br />2018-02-12 16:46:31,151+01 INFO [= org.ovirt.engine.core.vdsbro<wbr />ker.monitoring.VmDevicesMonito<wbr /= troller=3D0, type=3Dvirtio-serial, port=3D1}', managed=3D'false', plugg= ed=3D'true', readOnly=3D'false', deviceAlias=3D'channel0', customProper= ties=3D'[]', snapshotId=3D'null', logicalName=3D'null', hostDevice=3D'n= ull'}, device=5Ffbddd528-7d93-49c6-a286<wbr />-180e021cb274device=5F879= c93ab-<wbr />4df1-435c-af02-565039fcc254dev<wbr />ice=5F8945f61a-abbe-4= 156-8485-a4<wbr />aa6f1908dbdevice=5F017b5e59-01c4<wbr />-4aac-bf0c-b5d= 9557284d6=3DVmDevi<wbr />ce:{id=3D'VmDeviceId:{deviceId=3D'<wbr />017b5= e59-01c4-4aac-bf0c-<wbr />b5d9557284d6', vmId=3D'3f57e669-5e4c-4d10-85c= c-<wbr />d573004a099d'}', device=3D'tablet', type=3D'UNKNOWN', bootOrde= r=3D'0', specParams=3D'[]', address=3D'{bus=3D0, type=3Dusb, port=3D1}'= , managed=3D'false', plugged=3D'true', readOnly=3D'false', deviceAlias=3D= 'input0', customProperties=3D'[]', snapshotId=3D'null', logicalName=3D'= null', hostDevice=3D'null'}, device=5Ffbddd528-7d93-49c6-a286<wbr />-18= 0e021cb274=3DVmDevice:{id=3D'<wbr />VmDeviceId:{deviceId=3D'fbddd528<wb= r />-7d93-49c6-a286-180e021cb274', vmId=3D'3f57e669-5e4c-4d10-85cc-<wbr= />d573004a099d'}', device=3D'ide', type=3D'CONTROLLER', bootOrder=3D'0= ', specParams=3D'[]', address=3D'{slot=3D0x01, bus=3D0x00, domain=3D0x0= 000, type=3Dpci, function=3D0x1}', managed=3D'false', plugged=3D'true',= readOnly=3D'false', deviceAlias=3D'ide', customProperties=3D'[]', snap= shotId=3D'null', logicalName=3D'null', hostDevice=3D'null'}, device=5Ff= bddd528-7d93-49c6-a286<wbr />-180e021cb274device=5F879c93ab-<wbr />4df1= -435c-af02-565039fcc254dev<wbr />ice=5F8945f61a-abbe-4156-8485-<wbr />a= 4aa6f1908db=3DVmDevice:{id=3D'<wbr />VmDeviceId:{deviceId=3D'<wbr />894= 5f61a-abbe-4156-8485-<wbr />a4aa6f1908db', vmId=3D'3f57e669-5e4c-4d10-8= 5cc-<wbr />d573004a099d'}', device=3D'unix', type=3D'CHANNEL', bootOrde= r=3D'0', specParams=3D'[]', address=3D'{bus=3D0, controller=3D0, type=3D= virtio-serial, port=3D2}', managed=3D'false', plugged=3D'true', readOnl= y=3D'false', deviceAlias=3D'channel1', customProperties=3D'[]', snapsho= tId=3D'null', logicalName=3D'null', hostDevice=3D'null'}}, vmType=3Dkvm= , memSize=3D8192, smpCoresPerSocket=3D1, vmName=3DOracle=5FSECONDARY, n= ice=3D0, status=3DUp, maxMemSize=3D32768, bootMenuEnable=3Dfalse, vmId=3D= 3f57e669-5e4c-4d10-85cc-d<wbr />573004a099d, numOfIoThreads=3D2, smpThr= eadsPerCore=3D1, smartcardEnable=3Dfalse, maxMemSlots=3D16, kvmEnable=3D= true, pitReinjection=3Dfalse, displayNetwork=3Dovirtmgmt, devices=3D[Lj= ava.lang.Object;@2e<wbr />4d3dd3, memGuaranteedSize=3D8192, maxVCpus=3D= 16, clientIp=3D, statusTime=3D<a value=3D"+14304259600" target=3D"=5Fbl= ank" href=3D"tel:(430)%20425-9600">4304259600</a>, display=3Dvnc}], log= id: 7cc65298<br />2018-02-12 16:46:42,257+01 INFO [org.ovirt.eng= ine.core.vdsbro<wbr />ker.monitoring.VmDevicesMonito<wbr />ring] (ForkJ= oinPool-1-worker-4) [] Received a vnc Device without an address when pr= ocessing VM 3f57e669-5e4c-4d10-85cc-d57300<wbr />4a099d devices, skippi= ng device: {device=3Dvnc, specParams=3D{displayNetwork=3Dovi<wbr />rtmg= mt, keyMap=3Dfr, displayIp=3D192.168.0.5}, type=3Dgraphics, deviceId=3D= 813957b1-446a-4e88-9e<wbr />40-9fe76d2c442d, port=3D5901}<br />2018-02-= 12 16:46:42,257+01 INFO [org.ovirt.engine.core.vdsbro<wbr />ker.m= onitoring.VmDevicesMonito<wbr />ring] (ForkJoinPool-1-worker-4) [] Rece= ived a lease Device without an address when processing VM 3f57e669-5e4c= -4d10-85cc-d57300<wbr />4a099d devices, skipping device: {lease=5Fid=3D= 3f57e669-5e4c-4d10-8<wbr />5cc-d573004a099d, sd=5Fid=3D1e51cecc-eb2e-47= d0-b185-<wbr />920fdc7afa16, deviceId=3D{uuid=3Da09949aa-5642-4<wbr />b= 6d-94a4-8b0d04257be5}, offset=3D6291456, device=3Dlease, path=3D/rhev/d= ata-center/mnt/glu<wbr />sterSD/192.168.0.6:=5FDATA01/1e5<wbr />1cecc-e= b2e-47d0-b185-920fdc7af<wbr />a16/dom=5Fmd/xleases, type=3Dlease}<br />= 2018-02-12 16:46:46,260+01 INFO [org.ovirt.engine.core.vdsbro<wbr= />ker.vdsbroker.FullListVDSComma<wbr />nd] (DefaultQuartzScheduler5) [= 7fcb200a] FINISH, FullListVDSCommand, return: [{acpiEnable=3Dtrue, emul= atedMachine=3Dpc-i440fx-rhel<wbr />7.3.0, afterMigrationStatus=3D, tabl= etEnable=3Dtrue, pid=3D18748, guestDiskMapping=3D{0QEMU=5FQEMU=5FH<wbr = />ARDDISK=5Fd890fa68-fba4-4f49-9=3D{<wbr />name=3D/dev/sda}, QEMU=5FDVD= -ROM=5FQM00003=3D{name=3D/de<wbr />v/sr0}}, transparentHugePages=3Dtrue= , timeOffset=3D0, cpuType=3DNehalem, smp=3D2, guestNumaNodes=3D[Ljava.l= ang.Obj<wbr />ect;@77951faf, custom=3D{device=5Ffbddd528-7d93-4<wbr />9= c6-a286-180e021cb274device=5F87<wbr />9c93ab-4df1-435c-af02-565039fc<wb= r />c254=3DVmDevice:{id=3D'VmDeviceId:<wbr />{deviceId=3D'879c93ab-4df1= -435c-<wbr />af02-565039fcc254', vmId=3D'3f57e669-5e4c-4d10-85cc-<wbr /= pecParams=3D'[]', address=3D'{bus=3D0, controller=3D0, type=3Dvirtio-se= rial, port=3D1}', managed=3D'false', plugged=3D'true', readOnly=3D'fals= e', deviceAlias=3D'channel0', customProperties=3D'[]', snapshotId=3D'nu= ll', logicalName=3D'null', hostDevice=3D'null'}, device=5Ffbddd528-7d93= -49c6-a286<wbr />-180e021cb274device=5F879c93ab-<wbr />4df1-435c-af02-5= 65039fcc254dev<wbr />ice=5F8945f61a-abbe-4156-8485-a4<wbr />aa6f1908dbd= evice=5F017b5e59-01c4<wbr />-4aac-bf0c-b5d9557284d6=3DVmDevi<wbr />ce:{= id=3D'VmDeviceId:{deviceId=3D'<wbr />017b5e59-01c4-4aac-bf0c-<wbr />b5d= 9557284d6', vmId=3D'3f57e669-5e4c-4d10-85cc-<wbr />d573004a099d'}', dev= ice=3D'tablet', type=3D'UNKNOWN', bootOrder=3D'0', specParams=3D'[]', a= ddress=3D'{bus=3D0, type=3Dusb, port=3D1}', managed=3D'false', plugged=3D= 'true', readOnly=3D'false', deviceAlias=3D'input0', customProperties=3D= '[]', snapshotId=3D'null', logicalName=3D'null', hostDevice=3D'null'}, = device=5Ffbddd528-7d93-49c6-a286<wbr />-180e021cb274=3DVmDevice:{id=3D'= <wbr />VmDeviceId:{deviceId=3D'fbddd528<wbr />-7d93-49c6-a286-180e021cb= 274', vmId=3D'3f57e669-5e4c-4d10-85cc-<wbr />d573004a099d'}', device=3D= 'ide', type=3D'CONTROLLER', bootOrder=3D'0', specParams=3D'[]', address= =3D'{slot=3D0x01, bus=3D0x00, domain=3D0x0000, type=3Dpci, function=3D0= x1}', managed=3D'false', plugged=3D'true', readOnly=3D'false', deviceAl= ias=3D'ide', customProperties=3D'[]', snapshotId=3D'null', logicalName=3D= 'null', hostDevice=3D'null'}, device=5Ffbddd528-7d93-49c6-a286<wbr />-1= 80e021cb274device=5F879c93ab-<wbr />4df1-435c-af02-565039fcc254dev<wbr = />ice=5F8945f61a-abbe-4156-8485-<wbr />a4aa6f1908db=3DVmDevice:{id=3D'<= wbr />VmDeviceId:{deviceId=3D'<wbr />8945f61a-abbe-4156-8485-<wbr />a4a= a6f1908db', vmId=3D'3f57e669-5e4c-4d10-85cc-<wbr />d573004a099d'}', dev= ice=3D'unix', type=3D'CHANNEL', bootOrder=3D'0', specParams=3D'[]', add= ress=3D'{bus=3D0, controller=3D0, type=3Dvirtio-serial, port=3D2}', man= aged=3D'false', plugged=3D'true', readOnly=3D'false', deviceAlias=3D'ch= annel1', customProperties=3D'[]', snapshotId=3D'null', logicalName=3D'n= ull', hostDevice=3D'null'}}, vmType=3Dkvm, memSize=3D8192, smpCoresPerS= ocket=3D1, vmName=3DOracle=5FSECONDARY, nice=3D0, status=3DUp, maxMemSi= ze=3D32768, bootMenuEnable=3Dfalse, vmId=3D3f57e669-5e4c-4d10-85cc-d<wb= r />573004a099d, numOfIoThreads=3D2, smpThreadsPerCore=3D1, smartcardEn= able=3Dfalse, maxMemSlots=3D16, kvmEnable=3Dtrue, pitReinjection=3Dfals= e, displayNetwork=3Dovirtmgmt, devices=3D[Ljava.lang.Object;@28<wbr />6= 410fd, memGuaranteedSize=3D8192, maxVCpus=3D16, clientIp=3D, statusTime= =3D<a value=3D"+14304263620" target=3D"=5Fblank" href=3D"tel:(430)%2042= 6-3620">4304263620</a>, display=3Dvnc}], log id: 58cdef4c<br />2018-02-= 12 16:46:46,267+01 INFO [org.ovirt.engine.core.vdsbro<wbr />ker.m= onitoring.VmDevicesMonito<wbr />ring] (DefaultQuartzScheduler5) [7fcb20= 0a] Received a vnc Device without an address when processing VM 3f57e66= 9-5e4c-4d10-85cc-d57300<wbr />4a099d devices, skipping device: {device=3D= vnc, specParams=3D{displayNetwork=3Dovi<wbr />rtmgmt, keyMap=3Dfr, disp= layIp=3D192.168.0.5}, type=3Dgraphics, deviceId=3D813957b1-446a-4e88-9e= <wbr />40-9fe76d2c442d, port=3D5901}<br />2018-02-12 16:46:46,268+01 IN= FO [org.ovirt.engine.core.vdsbro<wbr />ker.monitoring.VmDevicesMo= nito<wbr />ring] (DefaultQuartzScheduler5) [7fcb200a] Received a lease = Device without an address when processing VM 3f57e669-5e4c-4d10-85cc-d5= 7300<wbr />4a099d devices, skipping device: {lease=5Fid=3D3f57e669-5e4c= -4d10-8<wbr />5cc-d573004a099d, sd=5Fid=3D1e51cecc-eb2e-47d0-b185-<wbr = />920fdc7afa16, deviceId=3D{uuid=3Da09949aa-5642-4<wbr />b6d-94a4-8b0d0= 4257be5}, offset=3D6291456, device=3Dlease, path=3D/rhev/data-center/mn= t/glu<wbr />sterSD/192.168.0.6:=5FDATA01/1e5<wbr />1cecc-eb2e-47d0-b185= -920fdc7af<wbr />a16/dom=5Fmd/xleases, type=3Dlease}<p> </p></bloc= kquote><br />For the VM with 2 vdisks we see :<blockquote><p>2018-02-12= 16:49:06,112+01 INFO [org.ovirt.engine.core.bll.Mi<wbr />grateVm= ToServerCommand] (default task-50) [92b5af33-cb87-4142-b8fe-8b838<wbr /=
dd7458e] Lock Acquired to object 'EngineLock:{exclusiveLocks=3D'[<wbr = />f7d4ec12-627a-4b83-b59e-886400<wbr />d55474=3DVM]', sharedLocks=3D''}= '<br />2018-02-12 16:49:06,407+01 INFO [org.ovirt.engine.core.bll= .Mi<wbr />grateVmToServerCommand] (org.ovirt.thread.pool-6-threa<wbr />= d-49) [92b5af33-cb87-4142-b8fe-8b838<wbr />dd7458e] Running command: Mi= grateVmToServerCommand internal: false. Entities affected : ID: f= 7d4ec12-627a-4b83-b59e-886400<wbr />d55474 Type: VMAction group MIGRATE= =5FVM with role type USER<br />2018-02-12 16:49:06,712+01 INFO [o= rg.ovirt.engine.core.vdsbro<wbr />ker.MigrateVDSCommand] (org.ovirt.thr= ead.pool-6-threa<wbr />d-49) [92b5af33-cb87-4142-b8fe-8b838<wbr />dd745= 8e] START, MigrateVDSCommand( MigrateVDSCommandParameters:{r<wbr />unAs= ync=3D'true', hostId=3D'd569c2dd-8f30-4878-8ae<wbr />a-858db285cf69', v= mId=3D'f7d4ec12-627a-4b83-b59e-<wbr />886400d55474', srcHost=3D'192.168= .0.5', dstVdsId=3D'ce3938b1-b23f-4d22-8<wbr />40a-f17d7cd87bb1', dstHos= t=3D'<a target=3D"=5Fblank" href=3D"http://192.168.0.6:54321">192.168.0= .6:54321</a>', migrationMethod=3D'ONLINE', tunnelMigration=3D'false', m= igrationDowntime=3D'0', autoConverge=3D'true', migrateCompressed=3D'fal= se', consoleAddress=3D'null', maxBandwidth=3D'500', enableGuestEvents=3D= 'true', maxIncomingMigrations=3D'2', maxOutgoingMigrations=3D'2', conve= rgenceSchedule=3D'[init=3D[{n<wbr />ame=3DsetDowntime, params=3D[100]}]= , stalling=3D[{limit=3D1, action=3D{name=3DsetDowntime, params=3D[150]}= }, {limit=3D2, action=3D{name=3DsetDowntime, params=3D[200]}}, {limit=3D= 3, action=3D{name=3DsetDowntime, params=3D[300]}}, {limit=3D4, action=3D= {name=3DsetDowntime, params=3D[400]}}, {limit=3D6, action=3D{name=3Dset= Downtime, params=3D[500]}}, {limit=3D-1, action=3D{name=3Dabort, params= =3D[]}}]]'}), log id: 3702a9e0<br />2018-02-12 16:49:06,713+01 INFO &nb= sp;[org.ovirt.engine.core.vdsbro<wbr />ker.vdsbroker.MigrateBrokerVDS<w= br />Command] (org.ovirt.thread.pool-6-threa<wbr />d-49) [92b5af33-cb87= -4142-b8fe-8b838<wbr />dd7458e] START, MigrateBrokerVDSCommand(HostNa<w= br />me =3D <a target=3D"=5Fblank" href=3D"http://ginger.local.systea.f= r">ginger.local.systea.fr</a>, MigrateVDSCommandParameters:{r<wbr />unA= sync=3D'true', hostId=3D'd569c2dd-8f30-4878-8ae<wbr />a-858db285cf69', = vmId=3D'f7d4ec12-627a-4b83-b59e-<wbr />886400d55474', srcHost=3D'192.16= 8.0.5', dstVdsId=3D'ce3938b1-b23f-4d22-8<wbr />40a-f17d7cd87bb1', dstHo= st=3D'<a target=3D"=5Fblank" href=3D"http://192.168.0.6:54321">192.168.= 0.6:54321</a>', migrationMethod=3D'ONLINE', tunnelMigration=3D'false', = migrationDowntime=3D'0', autoConverge=3D'true', migrateCompressed=3D'fa= lse', consoleAddress=3D'null', maxBandwidth=3D'500', enableGuestEvents=3D= 'true', maxIncomingMigrations=3D'2', maxOutgoingMigrations=3D'2', conve= rgenceSchedule=3D'[init=3D[{n<wbr />ame=3DsetDowntime, params=3D[100]}]= , stalling=3D[{limit=3D1, action=3D{name=3DsetDowntime, params=3D[150]}= }, {limit=3D2, action=3D{name=3DsetDowntime, params=3D[200]}}, {limit=3D= 3, action=3D{name=3DsetDowntime, params=3D[300]}}, {limit=3D4, action=3D= {name=3DsetDowntime, params=3D[400]}}, {limit=3D6, action=3D{name=3Dset= Downtime, params=3D[500]}}, {limit=3D-1, action=3D{name=3Dabort, params= =3D[]}}]]'}), log id: 1840069c<br />2018-02-12 16:49:06,724+01 INFO &nb= sp;[org.ovirt.engine.core.vdsbro<wbr />ker.vdsbroker.MigrateBrokerVDS<w= br />Command] (org.ovirt.thread.pool-6-threa<wbr />d-49) [92b5af33-cb87= -4142-b8fe-8b838<wbr />dd7458e] FINISH, MigrateBrokerVDSCommand, log id= : 1840069c<br />2018-02-12 16:49:06,732+01 INFO [org.ovirt.engine= .core.vdsbro<wbr />ker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thre= a<wbr />d-49) [92b5af33-cb87-4142-b8fe-8b838<wbr />dd7458e] FINISH, Mig= rateVDSCommand, return: MigratingFrom, log id: 3702a9e0<br />2018-02-12= 16:49:06,753+01 INFO [org.ovirt.engine.core.dal.db<wbr />broker.= auditloghandling.AuditL<wbr />ogDirector] (org.ovirt.thread.pool-6-thre= a<wbr />d-49) [92b5af33-cb87-4142-b8fe-8b838<wbr />dd7458e] EVENT=5FID:= VM=5FMIGRATION=5FSTART(62), Correlation ID: 92b5af33-cb87-4142-b8fe-8b= 838d<wbr />d7458e, Job ID: f4f54054-f7c8-4481-8eda-d5a15c<wbr />383061,= Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migra= tion started (VM: Oracle=5FPRIMARY, Source: <a target=3D"=5Fblank" href= =3D"http://ginger.local.systea.fr">ginger.local.systea.fr</a>, Destinat= ion: <a target=3D"=5Fblank" href=3D"http://victor.local.systea.fr">vict= or.local.systea.fr</a>, User: admin@internal-authz).<br />...<br />2018= -02-12 16:49:16,453+01 INFO [org.ovirt.engine.core.vdsbro<wbr />k= er.monitoring.VmsStatisticsFe<wbr />tcher] (DefaultQuartzScheduler4) [1= 62a5bc3] Fetched 2 VMs from VDS 'ce3938b1-b23f-4d22-840a-f17d7<wbr />cd= 87bb1'<br />2018-02-12 16:49:16,455+01 INFO [org.ovirt.engine.cor= e.vdsbro<wbr />ker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [16= 2a5bc3] VM 'f7d4ec12-627a-4b83-b59e-88640<wbr />0d55474'(Oracle=5FPRIMA= RY) was unexpectedly detected as 'MigratingTo' on VDS 'ce3938b1-b23f-4d= 22-840a-f17d7<wbr />cd87bb1'(<a target=3D"=5Fblank" href=3D"http://vict= or.local.systea.fr">victor.local.systea.<wbr />fr</a>) (expected on 'd5= 69c2dd-8f30-4878-8aea-858db<wbr />285cf69')<br />2018-02-12 16:49:16,45= 5+01 INFO [org.ovirt.engine.core.vdsbro<wbr />ker.monitoring.VmAn= alyzer] (DefaultQuartzScheduler4) [162a5bc3] VM 'f7d4ec12-627a-4b83-b59= e-88640<wbr />0d55474' is migrating to VDS 'ce3938b1-b23f-4d22-840a-f17= d7<wbr />cd87bb1'(<a target=3D"=5Fblank" href=3D"http://victor.local.sy= stea.fr">victor.local.systea.<wbr />fr</a>) ignoring it in the refresh = until migration is done<br />...<br />2018-02-12 16:49:31,484+01 INFO &= nbsp;[org.ovirt.engine.core.vdsbro<wbr />ker.monitoring.VmAnalyzer] (De= faultQuartzScheduler5) [11a7619a] VM 'f7d4ec12-627a-4b83-b59e-88640<wbr= />0d55474'(Oracle=5FPRIMARY) was unexpectedly detected as 'MigratingTo= ' on VDS 'ce3938b1-b23f-4d22-840a-f17d7<wbr />cd87bb1'(<a target=3D"=5F= blank" href=3D"http://victor.local.systea.fr">victor.local.systea.<wbr = />fr</a>) (expected on 'd569c2dd-8f30-4878-8aea-858db<wbr />285cf69')<b= r />2018-02-12 16:49:31,484+01 INFO [org.ovirt.engine.core.vdsbro= <wbr />ker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [11a7619a] = VM 'f7d4ec12-627a-4b83-b59e-88640<wbr />0d55474' is migrating to VDS 'c= e3938b1-b23f-4d22-840a-f17d7<wbr />cd87bb1'(<a target=3D"=5Fblank" href= =3D"http://victor.local.systea.fr">victor.local.systea.<wbr />fr</a>) i= gnoring it in the refresh until migration is done<br /> </p></bloc= kquote><br />and so on, last lines repeated indefinitly for hours since= we poweroff the VM...<br />Is this something known ? Any idea about th= at ?<br /><br />Thanks<br /><br />Ovirt 4.1.6, updated last at feb-13. = Gluster 3.12.1.<br /><br />--<p class=3D"m=5F-4299273321983674487m=5F85= 87729722327689770Text1">Cordialement,<br /><br /><b>Frank Soyer </b></p= <br />=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F<wbr />=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F<br />Users mailing list<br /><a target=3D"=5Fblank" href=3D"m= ailto:Users@ovirt.org">Users@ovirt.org</a><br /><a rel=3D"noreferrer" t= arget=3D"=5Fblank" href=3D"http://lists.ovirt.org/mailman/listinfo/user= s">http://lists.ovirt.org/mailman<wbr />/listinfo/users</a><br /> = </blockquote></div></div></blockquote><br /> </div></div></blockqu= ote></div></div></blockquote><br /> </html>
------=_=-_OpenGroupware_org_NGMime-18019-1519309377.654147-22--------