[Users] VM migrations failing
Engine Build --> Commit: 82bdc46dfdb46b000f67f0cd4e51fc39665bf13b VDSM Build: --> Commit: da89a27492cc7d5a84e4bb87652569ca8e0fb20e + patch --> http://gerrit.ovirt.org/#/c/11492/ Engine Side: 2013-01-30 10:56:38,439 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) Rerun vm 887d764a-f835-4112-9eda-836a772ea5eb. Called from vds lostisles 2013-01-30 10:56:38,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) START, MigrateStatusVDSCommand(HostName = lostisles, HostId = e042b03b-dd4e-414c-be1a-b2c65ac000f5, vmId=887d764a-f835-4112-9eda-836a772ea5eb), log id: 6556e75b 2013-01-30 10:56:38,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Failed in MigrateStatusVDS method 2013-01-30 10:56:38,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Error code migrateErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Fatal error during migration 2013-01-30 10:56:38,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12, mMessage=Fatal error during migration]] VDSM Side: Thread-43670::ERROR::2013-01-30 10:56:37,052::vm::200::vm.Vm::(_recover) vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::this function is not supported by the connection driver: virDomainMigrateToURI2 Thread-43670::ERROR::2013-01-30 10:56:37,513::vm::288::vm.Vm::(run) vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 273, in run self._startUnderlyingMigration() File "/usr/share/vdsm/libvirtvm.py", line 504, in _startUnderlyingMigration None, maxBandwidth) File "/usr/share/vdsm/libvirtvm.py", line 540, in f ret = attr(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1103, in migrateToURI2 if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self) libvirtError: this function is not supported by the connection driver: virDomainMigrateToURI2 GuestMonitor-sl63::DEBUG::2013-01-30 10:56:38,235::libvirtvm::307::vm.Vm::(_getDiskLatency) vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::Disk vda latency not available - DHC
On Wed, Jan 30, 2013 at 11:04:00AM -0600, Dead Horse wrote:
Engine Build --> Commit: 82bdc46dfdb46b000f67f0cd4e51fc39665bf13b VDSM Build: --> Commit: da89a27492cc7d5a84e4bb87652569ca8e0fb20e + patch --> http://gerrit.ovirt.org/#/c/11492/
Engine Side: 2013-01-30 10:56:38,439 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) Rerun vm 887d764a-f835-4112-9eda-836a772ea5eb. Called from vds lostisles 2013-01-30 10:56:38,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) START, MigrateStatusVDSCommand(HostName = lostisles, HostId = e042b03b-dd4e-414c-be1a-b2c65ac000f5, vmId=887d764a-f835-4112-9eda-836a772ea5eb), log id: 6556e75b 2013-01-30 10:56:38,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Failed in MigrateStatusVDS method 2013-01-30 10:56:38,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Error code migrateErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Fatal error during migration 2013-01-30 10:56:38,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12, mMessage=Fatal error during migration]]
VDSM Side: Thread-43670::ERROR::2013-01-30 10:56:37,052::vm::200::vm.Vm::(_recover) vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::this function is not supported by the connection driver: virDomainMigrateToURI2 Thread-43670::ERROR::2013-01-30 10:56:37,513::vm::288::vm.Vm::(run) vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 273, in run self._startUnderlyingMigration() File "/usr/share/vdsm/libvirtvm.py", line 504, in _startUnderlyingMigration None, maxBandwidth) File "/usr/share/vdsm/libvirtvm.py", line 540, in f ret = attr(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1103, in migrateToURI2 if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self) libvirtError: this function is not supported by the connection driver: virDomainMigrateToURI2
Could it be that you are using an ancient libvirt with no virDomainMigrateToURI2? What are your libvirt and qemu-kvm versions (on both machines)?
The nodes are EL6.3 based. Currently installed libvirt packages: libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 libvirt-cim-0.6.1-3.el6.x86_64 libvirt-0.9.10-21.el6_3.8.x86_64 libvirt-python-0.9.10-21.el6_3.8.x86_64 libvirt-client-0.9.10-21.el6_3.8.x86_64 and qemu packages: qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 qemu-img-0.12.1.2-2.295.el6_3.10.x86_64 Thus my presumption here given the above is that virDomainMigrateToURI2 has not yet been patched and/or back-ported into the EL6.x libvirt/qemu? - DHC On Wed, Jan 30, 2013 at 1:28 PM, Dan Kenigsberg <danken@redhat.com> wrote:
Engine Build --> Commit: 82bdc46dfdb46b000f67f0cd4e51fc39665bf13b VDSM Build: --> Commit: da89a27492cc7d5a84e4bb87652569ca8e0fb20e + patch --> http://gerrit.ovirt.org/#/c/11492/
Engine Side: 2013-01-30 10:56:38,439 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) Rerun vm 887d764a-f835-4112-9eda-836a772ea5eb. Called from vds lostisles 2013-01-30 10:56:38,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) START, MigrateStatusVDSCommand(HostName = lostisles, HostId = e042b03b-dd4e-414c-be1a-b2c65ac000f5, vmId=887d764a-f835-4112-9eda-836a772ea5eb), log id: 6556e75b 2013-01-30 10:56:38,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Failed in MigrateStatusVDS method 2013-01-30 10:56:38,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Error code migrateErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Fatal error during migration 2013-01-30 10:56:38,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12, mMessage=Fatal error during migration]]
VDSM Side: Thread-43670::ERROR::2013-01-30 10:56:37,052::vm::200::vm.Vm::(_recover) vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::this function is not supported by the connection driver: virDomainMigrateToURI2 Thread-43670::ERROR::2013-01-30 10:56:37,513::vm::288::vm.Vm::(run) vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 273, in run self._startUnderlyingMigration() File "/usr/share/vdsm/libvirtvm.py", line 504, in _startUnderlyingMigration None, maxBandwidth) File "/usr/share/vdsm/libvirtvm.py", line 540, in f ret = attr(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py",
On Wed, Jan 30, 2013 at 11:04:00AM -0600, Dead Horse wrote: line
111, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1103, in migrateToURI2 if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self) libvirtError: this function is not supported by the connection driver: virDomainMigrateToURI2
Could it be that you are using an ancient libvirt with no virDomainMigrateToURI2? What are your libvirt and qemu-kvm versions (on both machines)?
This is a multi-part message in MIME format. --------------040706080902050106070109 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, Just curiously, did you install oVirt packages from http://resources.ovirt.org/releases/nightly/rpm/EL/6/noarch/ ? And there is no stable oVirt release 3.1/3.2 for RHEL. Dead Horse:
The nodes are EL6.3 based.
Currently installed libvirt packages:
libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 libvirt-cim-0.6.1-3.el6.x86_64 libvirt-0.9.10-21.el6_3.8.x86_64 libvirt-python-0.9.10-21.el6_3.8.x86_64 libvirt-client-0.9.10-21.el6_3.8.x86_64
and qemu packages: qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
Thus my presumption here given the above is that virDomainMigrateToURI2 has not yet been patched and/or back-ported into the EL6.x libvirt/qemu?
virDomainMigrateToURI2() is supported in libvirt-0.9.10, I am not sure if qemu-kvm-0.12.1.2 supports it. /var/log/libvirtd.log may help to verify this assumption
- DHC
On Wed, Jan 30, 2013 at 1:28 PM, Dan Kenigsberg <danken@redhat.com <mailto:danken@redhat.com>> wrote:
On Wed, Jan 30, 2013 at 11:04:00AM -0600, Dead Horse wrote: > Engine Build --> Commit: 82bdc46dfdb46b000f67f0cd4e51fc39665bf13b > VDSM Build: --> Commit: da89a27492cc7d5a84e4bb87652569ca8e0fb20e + patch > --> http://gerrit.ovirt.org/#/c/11492/ > > Engine Side: > 2013-01-30 10:56:38,439 ERROR > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (QuartzScheduler_Worker-70) Rerun vm 887d764a-f835-4112-9eda-836a772ea5eb. > Called from vds lostisles > 2013-01-30 10:56:38,506 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (pool-3-thread-49) START, MigrateStatusVDSCommand(HostName = lostisles, > HostId = e042b03b-dd4e-414c-be1a-b2c65ac000f5, > vmId=887d764a-f835-4112-9eda-836a772ea5eb), log id: 6556e75b > 2013-01-30 10:56:38,510 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (pool-3-thread-49) Failed in MigrateStatusVDS method > 2013-01-30 10:56:38,510 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (pool-3-thread-49) Error code migrateErr and error message > VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = > Fatal error during migration > 2013-01-30 10:56:38,511 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (pool-3-thread-49) Command > org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return > value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12, > mMessage=Fatal error during migration]] > > > VDSM Side: > Thread-43670::ERROR::2013-01-30 10:56:37,052::vm::200::vm.Vm::(_recover) > vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::this function is not supported > by the connection driver: virDomainMigrateToURI2 > Thread-43670::ERROR::2013-01-30 10:56:37,513::vm::288::vm.Vm::(run) > vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::Failed to migrate > Traceback (most recent call last): > File "/usr/share/vdsm/vm.py", line 273, in run > self._startUnderlyingMigration() > File "/usr/share/vdsm/libvirtvm.py", line 504, in > _startUnderlyingMigration > None, maxBandwidth) > File "/usr/share/vdsm/libvirtvm.py", line 540, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line > 111, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1103, in > migrateToURI2 > if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', > dom=self) > libvirtError: this function is not supported by the connection driver: > virDomainMigrateToURI2
Could it be that you are using an ancient libvirt with no virDomainMigrateToURI2? What are your libvirt and qemu-kvm versions (on both machines)?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- --- ?? Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming@cn.ibm.com or shuming@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC --------------040706080902050106070109 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">Hi,<br> <br> Just curiously, did you install oVirt packages from <a class="moz-txt-link-freetext" href="http://resources.ovirt.org/releases/nightly/rpm/EL/6/noarch/">http://resources.ovirt.org/releases/nightly/rpm/EL/6/noarch/</a> ; ?<br> And there is no stable oVirt release 3.1/3.2 for RHEL. <br> <br> Dead Horse:<br> </div> <blockquote cite="mid:CAEWPe=p8TOpwr66DMcWyLmJ+UK5ky5vVSCVFRJBzOqRgEeqxeQ@mail.gmail.com" type="cite"> <div dir="ltr"> <div>The nodes are EL6.3 based.<br> <br> Currently installed libvirt packages:<br> <br> libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64<br> libvirt-cim-0.6.1-3.el6.x86_64<br> libvirt-0.9.10-21.el6_3.8.x86_64<br> libvirt-python-0.9.10-21.el6_3.8.x86_64<br> libvirt-client-0.9.10-21.el6_3.8.x86_64<br> <br> and qemu packages:<br> qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64<br> qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64<br> qemu-img-0.12.1.2-2.295.el6_3.10.x86_64<br> <br> </div> <div>Thus my presumption here given the above is that virDomainMigrateToURI2 has not yet been patched and/or back-ported into the EL6.x libvirt/qemu?<br> </div> </div> </blockquote> <br> virDomainMigrateToURI2() is supported in libvirt-0.9.10, I am not sure if qemu-kvm-0.12.1.2 supports it.<br> /var/log/libvirtd.log may help to verify this assumption<br> <br> <blockquote cite="mid:CAEWPe=p8TOpwr66DMcWyLmJ+UK5ky5vVSCVFRJBzOqRgEeqxeQ@mail.gmail.com" type="cite"> <div dir="ltr"> <div> </div> <div><br> </div> - DHC<br> </div> <div class="gmail_extra"><br> <br> <div class="gmail_quote">On Wed, Jan 30, 2013 at 1:28 PM, Dan Kenigsberg <span dir="ltr"><<a moz-do-not-send="true" href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class="HOEnZb"> <div class="h5">On Wed, Jan 30, 2013 at 11:04:00AM -0600, Dead Horse wrote:<br> > Engine Build --> Commit: 82bdc46dfdb46b000f67f0cd4e51fc39665bf13b<br> > VDSM Build: --> Commit: da89a27492cc7d5a84e4bb87652569ca8e0fb20e + patch<br> > --> <a moz-do-not-send="true" href="http://gerrit.ovirt.org/#/c/11492/" target="_blank">http://gerrit.ovirt.org/#/c/11492/</a><br> ><br> > Engine Side:<br> > 2013-01-30 10:56:38,439 ERROR<br> > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> > (QuartzScheduler_Worker-70) Rerun vm 887d764a-f835-4112-9eda-836a772ea5eb.<br> > Called from vds lostisles<br> > 2013-01-30 10:56:38,506 INFO<br> > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> > (pool-3-thread-49) START, MigrateStatusVDSCommand(HostName = lostisles,<br> > HostId = e042b03b-dd4e-414c-be1a-b2c65ac000f5,<br> > vmId=887d764a-f835-4112-9eda-836a772ea5eb), log id: 6556e75b<br> > 2013-01-30 10:56:38,510 ERROR<br> > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> > (pool-3-thread-49) Failed in MigrateStatusVDS method<br> > 2013-01-30 10:56:38,510 ERROR<br> > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> > (pool-3-thread-49) Error code migrateErr and error message<br> > VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error =<br> > Fatal error during migration<br> > 2013-01-30 10:56:38,511 INFO<br> > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> > (pool-3-thread-49) Command<br> > org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return<br> > value<br> > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12,<br> > mMessage=Fatal error during migration]]<br> ><br> ><br> > VDSM Side:<br> > Thread-43670::ERROR::2013-01-30 10:56:37,052::vm::200::vm.Vm::(_recover)<br> > vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::this function is not supported<br> > by the connection driver: virDomainMigrateToURI2<br> > Thread-43670::ERROR::2013-01-30 10:56:37,513::vm::288::vm.Vm::(run)<br> > vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::Failed to migrate<br> > Traceback (most recent call last):<br> > File "/usr/share/vdsm/vm.py", line 273, in run<br> > self._startUnderlyingMigration()<br> > File "/usr/share/vdsm/libvirtvm.py", line 504, in<br> > _startUnderlyingMigration<br> > None, maxBandwidth)<br> > File "/usr/share/vdsm/libvirtvm.py", line 540, in f<br> > ret = attr(*args, **kwargs)<br> > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line<br> > 111, in wrapper<br> > ret = f(*args, **kwargs)<br> > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1103, in<br> > migrateToURI2<br> > if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed',<br> > dom=self)<br> > libvirtError: this function is not supported by the connection driver:<br> > virDomainMigrateToURI2<br> <br> </div> </div> Could it be that you are using an ancient libvirt with no<br> virDomainMigrateToURI2? What are your libvirt and qemu-kvm versions (on<br> both machines)?<br> </blockquote> </div> <br> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre class="moz-signature" cols="72">-- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: <a class="moz-txt-link-abbreviated" href="mailto:shuming@cn.ibm.com">shuming@cn.ibm.com</a> or <a class="moz-txt-link-abbreviated" href="mailto:shuming@linux.vnet.ibm.com">shuming@linux.vnet.ibm.com</a> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC</pre> </body> </html> --------------040706080902050106070109--
On 01/30/2013 08:40 PM, Dead Horse wrote:
The nodes are EL6.3 based.
Currently installed libvirt packages:
libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 libvirt-cim-0.6.1-3.el6.x86_64 libvirt-0.9.10-21.el6_3.8.x86_64 libvirt-python-0.9.10-21.el6_3.8.x86_64 libvirt-client-0.9.10-21.el6_3.8.x86_64
and qemu packages: qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
Thus my presumption here given the above is that virDomainMigrateToURI2 has not yet been patched and/or back-ported into the EL6.x libvirt/qemu?
virDomainMigrateToURI2 is supported since 0.9.2, but is there a possibility the code is requesting direct migration? That might explain the message, which is then incorrect; this was fixed in [1]. Martin [1] http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc...
- DHC
On Wed, Jan 30, 2013 at 1:28 PM, Dan Kenigsberg <danken@redhat.com> wrote:
Engine Build --> Commit: 82bdc46dfdb46b000f67f0cd4e51fc39665bf13b VDSM Build: --> Commit: da89a27492cc7d5a84e4bb87652569ca8e0fb20e + patch --> http://gerrit.ovirt.org/#/c/11492/
Engine Side: 2013-01-30 10:56:38,439 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) Rerun vm 887d764a-f835-4112-9eda-836a772ea5eb. Called from vds lostisles 2013-01-30 10:56:38,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) START, MigrateStatusVDSCommand(HostName = lostisles, HostId = e042b03b-dd4e-414c-be1a-b2c65ac000f5, vmId=887d764a-f835-4112-9eda-836a772ea5eb), log id: 6556e75b 2013-01-30 10:56:38,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Failed in MigrateStatusVDS method 2013-01-30 10:56:38,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Error code migrateErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Fatal error during migration 2013-01-30 10:56:38,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12, mMessage=Fatal error during migration]]
VDSM Side: Thread-43670::ERROR::2013-01-30 10:56:37,052::vm::200::vm.Vm::(_recover) vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::this function is not supported by the connection driver: virDomainMigrateToURI2 Thread-43670::ERROR::2013-01-30 10:56:37,513::vm::288::vm.Vm::(run) vmId=`887d764a-f835-4112-9eda-836a772ea5eb`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 273, in run self._startUnderlyingMigration() File "/usr/share/vdsm/libvirtvm.py", line 504, in _startUnderlyingMigration None, maxBandwidth) File "/usr/share/vdsm/libvirtvm.py", line 540, in f ret = attr(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py",
On Wed, Jan 30, 2013 at 11:04:00AM -0600, Dead Horse wrote: line
111, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1103, in migrateToURI2 if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self) libvirtError: this function is not supported by the connection driver: virDomainMigrateToURI2
Could it be that you are using an ancient libvirt with no virDomainMigrateToURI2? What are your libvirt and qemu-kvm versions (on both machines)?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
On 01/30/2013 08:40 PM, Dead Horse wrote:
The nodes are EL6.3 based.
Currently installed libvirt packages:
libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 libvirt-cim-0.6.1-3.el6.x86_64 libvirt-0.9.10-21.el6_3.8.x86_64 libvirt-python-0.9.10-21.el6_3.8.x86_64 libvirt-client-0.9.10-21.el6_3.8.x86_64
and qemu packages: qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
Thus my presumption here given the above is that virDomainMigrateToURI2 has not yet been patched and/or back-ported into the EL6.x libvirt/qemu?
virDomainMigrateToURI2 is supported since 0.9.2, but is there a possibility the code is requesting direct migration? That might explain the message, which is then incorrect; this was fixed in [1].
Martin
[1] http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc...
What is "direct migration" exactly, in the context of qemu-kvm? We are using p2p migration http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140...
On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
On 01/30/2013 08:40 PM, Dead Horse wrote:
The nodes are EL6.3 based.
Currently installed libvirt packages:
libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 libvirt-cim-0.6.1-3.el6.x86_64 libvirt-0.9.10-21.el6_3.8.x86_64 libvirt-python-0.9.10-21.el6_3.8.x86_64 libvirt-client-0.9.10-21.el6_3.8.x86_64
and qemu packages: qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
Thus my presumption here given the above is that virDomainMigrateToURI2 has not yet been patched and/or back-ported into the EL6.x libvirt/qemu?
virDomainMigrateToURI2 is supported since 0.9.2, but is there a possibility the code is requesting direct migration? That might explain the message, which is then incorrect; this was fixed in [1].
Martin
[1] http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc...
What is "direct migration" exactly, in the context of qemu-kvm?
We are using p2p migration http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140...
OK, so that's not the issue, sorry for the confusion. I was thinking it would "somehow" get there. Direct migration doesn't exist in QEMU at all, so it seemed weird, but I can't seem to find any other reason for this failure; will keep searching, though. Martin
On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote:
On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
On 01/30/2013 08:40 PM, Dead Horse wrote:
The nodes are EL6.3 based.
Currently installed libvirt packages:
libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 libvirt-cim-0.6.1-3.el6.x86_64 libvirt-0.9.10-21.el6_3.8.x86_64 libvirt-python-0.9.10-21.el6_3.8.x86_64 libvirt-client-0.9.10-21.el6_3.8.x86_64
and qemu packages: qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
Thus my presumption here given the above is that virDomainMigrateToURI2 has not yet been patched and/or back-ported into the EL6.x libvirt/qemu?
virDomainMigrateToURI2 is supported since 0.9.2, but is there a possibility the code is requesting direct migration? That might explain the message, which is then incorrect; this was fixed in [1].
Martin
[1] http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc...
What is "direct migration" exactly, in the context of qemu-kvm?
We are using p2p migration http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140...
OK, so that's not the issue, sorry for the confusion. I was thinking it would "somehow" get there. Direct migration doesn't exist in QEMU at all, so it seemed weird, but I can't seem to find any other reason for this failure; will keep searching, though.
In this case, Dead Horse, would you try to migrate a VM (that you do not care much about) using virsh -c qemu+tls://hostname/system migrate --p2p dsthost? I'd like to see that the problem reproduces this way, too. More of libvirtd.log may help. You may want to disable iptables for a moment, just to eliminate a common cause of failure.
On 01/31/2013 11:56 AM, Dan Kenigsberg wrote:
On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote:
On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
On 01/30/2013 08:40 PM, Dead Horse wrote:
The nodes are EL6.3 based.
Currently installed libvirt packages:
libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 libvirt-cim-0.6.1-3.el6.x86_64 libvirt-0.9.10-21.el6_3.8.x86_64 libvirt-python-0.9.10-21.el6_3.8.x86_64 libvirt-client-0.9.10-21.el6_3.8.x86_64
and qemu packages: qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
Thus my presumption here given the above is that virDomainMigrateToURI2 has not yet been patched and/or back-ported into the EL6.x libvirt/qemu?
virDomainMigrateToURI2 is supported since 0.9.2, but is there a possibility the code is requesting direct migration? That might explain the message, which is then incorrect; this was fixed in [1].
Martin
[1] http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc...
What is "direct migration" exactly, in the context of qemu-kvm?
We are using p2p migration http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140...
OK, so that's not the issue, sorry for the confusion. I was thinking it would "somehow" get there. Direct migration doesn't exist in QEMU at all, so it seemed weird, but I can't seem to find any other reason for this failure; will keep searching, though.
In this case, Dead Horse, would you try to migrate a VM (that you do not care much about) using virsh -c qemu+tls://hostname/system migrate --p2p dsthost?
I'd like to see that the problem reproduces this way, too. More of libvirtd.log may help. You may want to disable iptables for a moment, just to eliminate a common cause of failure.
The error message in this version of libvirt is emitted only in two cases. Either the QEMU driver doesn't support peer2peer migration (which it does) or direct migration was requested (which wasn't) :) So I agree with Dan, please try to reproduce this without ovirt/vdsm and let us know (logs [1] will help a lot), because in case nobody fiddled with anything, this sounds like a bug to me. Martin [1] http://libvirt.org/logging.html
Shu, I build oVirt Engine and vdsm from source myself. The commits I indicated are what I built from.I run the engine under FC17 and my nodes are running EL6.x respectively. Dan, I reverted VDSM on my two test nodes to an earlier build of VDSM (commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4) VDSM after the above commit is broken due to commit: fc3a44f71d2ef202cff18d7203b9e4165b546621 however when I built and tested from master yesterday I did apply a patch I tested for ybronhei which fixed that issue. I will build VDSM from master, today w/ the supervdsm patch and try the manual migration you indicated. - DHC On Thu, Jan 31, 2013 at 4:56 AM, Dan Kenigsberg <danken@redhat.com> wrote:
On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote:
On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
On 01/30/2013 08:40 PM, Dead Horse wrote:
The nodes are EL6.3 based.
Currently installed libvirt packages:
libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 libvirt-cim-0.6.1-3.el6.x86_64 libvirt-0.9.10-21.el6_3.8.x86_64 libvirt-python-0.9.10-21.el6_3.8.x86_64 libvirt-client-0.9.10-21.el6_3.8.x86_64
and qemu packages: qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
Thus my presumption here given the above is that virDomainMigrateToURI2 has not yet been patched and/or back-ported into the EL6.x libvirt/qemu?
virDomainMigrateToURI2 is supported since 0.9.2, but is there a possibility the code is requesting direct migration? That might explain the message, which is then incorrect; this was fixed in [1].
Martin
[1]
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc...
What is "direct migration" exactly, in the context of qemu-kvm?
We are using p2p migration
http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140...
OK, so that's not the issue, sorry for the confusion. I was thinking it would "somehow" get there. Direct migration doesn't exist in QEMU at all, so it seemed weird, but I can't seem to find any other reason for this failure; will keep searching, though.
In this case, Dead Horse, would you try to migrate a VM (that you do not care much about) using virsh -c qemu+tls://hostname/system migrate --p2p dsthost?
I'd like to see that the problem reproduces this way, too. More of libvirtd.log may help. You may want to disable iptables for a moment, just to eliminate a common cause of failure.
Doh, brain fart VDSM is not involved here for the purposed of the needed test. Here is my initial whack at it: Source Node: virsh # list Id Name State ---------------------------------------------------- 1 sl63 running virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system error: operation failed: Failed to connect to remote libvirt URI qemu+ssh:// 3.57.111.32/system virsh # migrate --live sl63 qemu+ssh://192.168.1.2/system The authenticity of host '3.57.111.32 (192.168.1.2)' can't be established. RSA key fingerprint is e5:1d:b3:e5:38:5f:e1:8b:73:26:9e:15:c8:0a:2d:ac. Are you sure you want to continue connecting (yes/no)? yes root@192.168.1.2's password: Please enter your authentication name: vdsm@ovirt Please enter your password: virsh # Dest Node After migrate --live: virsh # list Id Name State ---------------------------------------------------- 2 sl63 running virsh # On Thu, Jan 31, 2013 at 10:38 AM, Dead Horse <deadhorseconsulting@gmail.com>wrote:
Shu, I build oVirt Engine and vdsm from source myself. The commits I indicated are what I built from.I run the engine under FC17 and my nodes are running EL6.x respectively.
Dan, I reverted VDSM on my two test nodes to an earlier build of VDSM (commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4) VDSM after the above commit is broken due to commit: fc3a44f71d2ef202cff18d7203b9e4165b546621 however when I built and tested from master yesterday I did apply a patch I tested for ybronhei which fixed that issue.
I will build VDSM from master, today w/ the supervdsm patch and try the manual migration you indicated.
- DHC
On Thu, Jan 31, 2013 at 4:56 AM, Dan Kenigsberg <danken@redhat.com> wrote:
On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote:
On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
On 01/30/2013 08:40 PM, Dead Horse wrote:
The nodes are EL6.3 based.
Currently installed libvirt packages:
libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 libvirt-cim-0.6.1-3.el6.x86_64 libvirt-0.9.10-21.el6_3.8.x86_64 libvirt-python-0.9.10-21.el6_3.8.x86_64 libvirt-client-0.9.10-21.el6_3.8.x86_64
and qemu packages: qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
Thus my presumption here given the above is that virDomainMigrateToURI2 has not yet been patched and/or back-ported into the EL6.x libvirt/qemu?
virDomainMigrateToURI2 is supported since 0.9.2, but is there a possibility the code is requesting direct migration? That might explain the message, which is then incorrect; this was fixed in [1].
Martin
[1]
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc...
What is "direct migration" exactly, in the context of qemu-kvm?
We are using p2p migration
http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140...
OK, so that's not the issue, sorry for the confusion. I was thinking it would "somehow" get there. Direct migration doesn't exist in QEMU at all, so it seemed weird, but I can't seem to find any other reason for this failure; will keep searching, though.
In this case, Dead Horse, would you try to migrate a VM (that you do not care much about) using virsh -c qemu+tls://hostname/system migrate --p2p dsthost?
I'd like to see that the problem reproduces this way, too. More of libvirtd.log may help. You may want to disable iptables for a moment, just to eliminate a common cause of failure.
note ignore the IP diff in the ssh host auth --> copy/paste fail ;) - DHC On Thu, Jan 31, 2013 at 11:25 AM, Dead Horse <deadhorseconsulting@gmail.com>wrote:
Doh, brain fart VDSM is not involved here for the purposed of the needed test. Here is my initial whack at it:
Source Node:
virsh # list Id Name State ---------------------------------------------------- 1 sl63 running
virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system error: operation failed: Failed to connect to remote libvirt URI qemu+ssh://3.57.111.32/system
virsh # migrate --live sl63 qemu+ssh://192.168.1.2/system The authenticity of host '3.57.111.32 (192.168.1.2)' can't be established. RSA key fingerprint is e5:1d:b3:e5:38:5f:e1:8b:73:26:9e:15:c8:0a:2d:ac. Are you sure you want to continue connecting (yes/no)? yes root@192.168.1.2's password: Please enter your authentication name: vdsm@ovirt Please enter your password:
virsh #
Dest Node After migrate --live: virsh # list Id Name State ---------------------------------------------------- 2 sl63 running
virsh #
On Thu, Jan 31, 2013 at 10:38 AM, Dead Horse < deadhorseconsulting@gmail.com> wrote:
Shu, I build oVirt Engine and vdsm from source myself. The commits I indicated are what I built from.I run the engine under FC17 and my nodes are running EL6.x respectively.
Dan, I reverted VDSM on my two test nodes to an earlier build of VDSM (commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4) VDSM after the above commit is broken due to commit: fc3a44f71d2ef202cff18d7203b9e4165b546621 however when I built and tested from master yesterday I did apply a patch I tested for ybronhei which fixed that issue.
I will build VDSM from master, today w/ the supervdsm patch and try the manual migration you indicated.
- DHC
On Thu, Jan 31, 2013 at 4:56 AM, Dan Kenigsberg <danken@redhat.com>wrote:
On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
On 01/30/2013 08:40 PM, Dead Horse wrote: > The nodes are EL6.3 based. > > Currently installed libvirt packages: > > libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 > libvirt-cim-0.6.1-3.el6.x86_64 > libvirt-0.9.10-21.el6_3.8.x86_64 > libvirt-python-0.9.10-21.el6_3.8.x86_64 > libvirt-client-0.9.10-21.el6_3.8.x86_64 > > and qemu packages: > qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 > qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 > qemu-img-0.12.1.2-2.295.el6_3.10.x86_64 > > Thus my presumption here given the above is that virDomainMigrateToURI2 has > not yet been patched and/or back-ported into the EL6.x
On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote: libvirt/qemu?
>
virDomainMigrateToURI2 is supported since 0.9.2, but is there a possibility the code is requesting direct migration? That might explain the message, which is then incorrect; this was fixed in [1].
Martin
[1]
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc...
What is "direct migration" exactly, in the context of qemu-kvm?
We are using p2p migration
http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140...
OK, so that's not the issue, sorry for the confusion. I was thinking it would "somehow" get there. Direct migration doesn't exist in QEMU at all, so it seemed weird, but I can't seem to find any other reason for this failure; will keep searching, though.
In this case, Dead Horse, would you try to migrate a VM (that you do not care much about) using virsh -c qemu+tls://hostname/system migrate --p2p dsthost?
I'd like to see that the problem reproduces this way, too. More of libvirtd.log may help. You may want to disable iptables for a moment, just to eliminate a common cause of failure.
Here is the content exceprt from libvirtd.log for the command: virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system 2013-01-31 18:02:53.740+0000: 2832: debug : virDomainFree:2313 : dom=0x7f4f88000c80, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) 2013-01-31 18:02:53.743+0000: 2831: debug : virDomainLookupByName:2146 : conn=0x7f4f8c001d80, name=sl63 2013-01-31 18:02:53.743+0000: 2831: debug : virDomainFree:2313 : dom=0x7f4f84002150, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) 2013-01-31 18:02:53.747+0000: 2829: debug : virDrvSupportsFeature:1521 : conn=0x7f4f8c001d80, feature=4 2013-01-31 18:02:53.751+0000: 2826: debug : virDrvSupportsFeature:1521 : conn=0x7f4f8c001d80, feature=6 2013-01-31 18:02:53.754+0000: 2828: debug : virDomainMigratePerform3:6247 : dom=0x7f4f7c0d1430, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), xmlin=(null) cookiein=(nil), cookieinlen=0, cookieout=0x7f4f998cab30, cookieoutlen=0x7f4f998cab3c, dconnuri=qemu+ssh://192.168.1.2/system, uri=(null), flags=2, dname=(null), bandwidth=0 2013-01-31 18:02:53.755+0000: 2828: debug : qemuMigrationPerform:2821 : driver=0x7f4f8c0af820, conn=0x7f4f8c001d80, vm=0x7f4f7c0c7040, xmlin=(null), dconnuri=qemu+ssh://192.168.1.2/system<http://3.57.111.32/system>, uri=(null), cookiein=(null), cookieinlen=0, cookieout=0x7f4f998cab30, cookieoutlen=0x7f4f998cab3c, flags=2, dname=(null), resource=0, v3proto=1 2013-01-31 18:02:53.755+0000: 2828: debug : qemuDomainObjBeginJobInternal:758 : Starting async job: migration out 2013-01-31 18:02:53.779+0000: 2828: debug : qemuProcessAutoDestroyActive:4226 : vm=sl63 2013-01-31 18:02:53.779+0000: 2828: debug : qemuDriverCloseCallbackGet:605 : vm=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb, conn=(nil) 2013-01-31 18:02:53.779+0000: 2828: debug : qemuDriverCloseCallbackGet:611 : cb=(nil) 2013-01-31 18:02:53.779+0000: 2828: debug : doPeer2PeerMigrate:2528 : driver=0x7f4f8c0af820, sconn=0x7f4f8c001d80, vm=0x7f4f7c0c7040, xmlin=(null), dconnuri=qemu+ssh://192.168.1.2/system<http://3.57.111.32/system>, uri=(null), flags=2, dname=(null), resource=0 2013-01-31 18:02:53.779+0000: 2828: debug : virConnectOpen:1349 : name=qemu+ssh://192.168.1.2/system <http://3.57.111.32/system> 2013-01-31 18:02:53.779+0000: 2828: debug : virConnectOpenResolveURIAlias:1070 : Loading config file '/etc/libvirt/libvirt.conf' 2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1151 : name "qemu+ssh:// 192.168.1.2/system <http://3.57.111.32/system>" to URI components: scheme qemu+ssh opaque (null) authority (null) server 192.168.1.2 user (null) port 0 path /system 2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 0 (Test) ... 2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1201 : driver 0 Test returned DECLINED 2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 1 (ESX) ... 2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1201 : driver 1 ESX returned DECLINED 2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 2 (remote) ... 2013-01-31 18:02:53.779+0000: 2828: debug : virCommandRunAsync:2174 : About to run LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin ssh 192.168.1.2 sh -c 'if 'nc' -q 2>&1 | grep "requires an argument" >/dev/null 2>&1; then ARG=-q0;else ARG=;fi;'nc' $ARG -U /var/run/libvirt/libvirt-sock' 2013-01-31 18:02:53.780+0000: 2828: debug : virCommandRunAsync:2192 : Command result 0, with PID 14537 2013-01-31 18:02:53.844+0000: 2828: error : virNetSocketReadWire:988 : Cannot recv data: Permission denied, please try again. Permission denied, please try again. : Connection reset by peerey,gssapi-keyex,gssapi-with-mic,password). 2013-01-31 18:02:53.845+0000: 2828: debug : do_open:1201 : driver 2 remote returned ERROR 2013-01-31 18:02:53.845+0000: 2828: error : doPeer2PeerMigrate:2539 : operation failed: Failed to connect to remote libvirt URI qemu+ssh:// 192.168.1.2/system <http://3.57.111.32/system> 2013-01-31 18:02:53.845+0000: 2828: debug : qemuDomainObjEndAsyncJob:888 : Stopping async job: migration out 2013-01-31 18:02:53.845+0000: 2821: debug : virPidAbort:2341 : aborting child process 14537 2013-01-31 18:02:53.845+0000: 2821: debug : virPidAbort:2346 : process has ended: exit status 255 2013-01-31 18:02:53.863+0000: 2828: debug : virDomainFree:2313 : dom=0x7f4f7c0d1430, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) 2013-01-31 18:02:54.459+0000: 2834: debug : virDomainInterfaceStats:7223 : dom=0x7f4f6c000910, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f95cc4b00, size=64 2013-01-31 18:02:54.459+0000: 2834: debug : virDomainFree:2313 : dom=0x7f4f6c000910, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) 2013-01-31 18:02:59.464+0000: 2825: debug : virDomainInterfaceStats:7223 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f9b6cdb00, size=64 2013-01-31 18:02:59.472+0000: 2825: debug : virDomainFree:2313 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) 2013-01-31 18:03:04.478+0000: 2829: debug : virDrvSupportsFeature:1521 : conn=0x7f4f7c000ae0, feature=9 2013-01-31 18:03:04.487+0000: 2826: debug : virDomainGetCPUStats:18676 : dom=0x7f4f70000dc0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), params=(nil), nparams=0, start_cpu=-1, ncpus=1, flags=4 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupNew:603 : New group /libvirt/qemu/sl63 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 0:cpu at /cgroup/cpu in 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 3:memory at /cgroup/memory in 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 4:devices at /cgroup/devices in 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 5:freezer at /cgroup/freezer in 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 6:blkio at /cgroup/blkio in 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:524 : Make group /libvirt/qemu/sl63 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpu/libvirt/qemu/sl63/ 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuacct/libvirt/qemu/sl63/ 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuset/libvirt/qemu/sl63/ 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/memory/libvirt/qemu/sl63/ 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/devices/libvirt/qemu/sl63/ 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/freezer/libvirt/qemu/sl63/ 2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/blkio/libvirt/qemu/sl63/ 2013-01-31 18:03:04.487+0000: 2826: debug : virDomainFree:2313 : dom=0x7f4f70000dc0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) 2013-01-31 18:03:04.491+0000: 2828: debug : virDrvSupportsFeature:1521 : conn=0x7f4f7c000ae0, feature=9 2013-01-31 18:03:04.495+0000: 2827: debug : virDomainGetCPUStats:18676 : dom=0x7f4f74000ce0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), params=0x7f4f74000b20, nparams=3, start_cpu=-1, ncpus=1, flags=4 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupNew:603 : New group /libvirt/qemu/sl63 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 0:cpu at /cgroup/cpu in 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 3:memory at /cgroup/memory in 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 4:devices at /cgroup/devices in 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 5:freezer at /cgroup/freezer in 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 6:blkio at /cgroup/blkio in 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:524 : Make group /libvirt/qemu/sl63 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpu/libvirt/qemu/sl63/ 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuacct/libvirt/qemu/sl63/ 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuset/libvirt/qemu/sl63/ 2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/memory/libvirt/qemu/sl63/ 2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/devices/libvirt/qemu/sl63/ 2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/freezer/libvirt/qemu/sl63/ 2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/blkio/libvirt/qemu/sl63/ 2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupGetValueStr:350 : Get value /cgroup/cpuacct/libvirt/qemu/sl63/cpuacct.usage 2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupGetValueStr:350 : Get value /cgroup/cpuacct/libvirt/qemu/sl63/cpuacct.stat 2013-01-31 18:03:04.496+0000: 2827: debug : virDomainFree:2313 : dom=0x7f4f74000ce0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) 2013-01-31 18:03:04.500+0000: 2825: debug : virDomainInterfaceStats:7223 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f9b6cdb00, size=64 2013-01-31 18:03:04.500+0000: 2825: debug : virDomainFree:2313 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) 2013-01-31 18:03:09.505+0000: 2829: debug : virDomainInterfaceStats:7223 : dom=0x7f4f800034a0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f98ec9b00, size=64 2013-01-31 18:03:09.520+0000: 2829: debug : virDomainFree:2313 : dom=0x7f4f800034a0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) On Thu, Jan 31, 2013 at 11:27 AM, Dead Horse <deadhorseconsulting@gmail.com>wrote:
note ignore the IP diff in the ssh host auth --> copy/paste fail ;) - DHC
On Thu, Jan 31, 2013 at 11:25 AM, Dead Horse < deadhorseconsulting@gmail.com> wrote:
Doh, brain fart VDSM is not involved here for the purposed of the needed test. Here is my initial whack at it:
Source Node:
virsh # list Id Name State ---------------------------------------------------- 1 sl63 running
virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system error: operation failed: Failed to connect to remote libvirt URI qemu+ssh://192.168.1.2/system <http://3.57.111.32/system>
virsh # migrate --live sl63 qemu+ssh://192.168.1.2/system The authenticity of host '192.168.1.2 (192.168.1.2)' can't be established. RSA key fingerprint is e5:1d:b3:e5:38:5f:e1:8b:73:26:9e:15:c8:0a:2d:ac. Are you sure you want to continue connecting (yes/no)? yes root@192.168.1.2's password: Please enter your authentication name: vdsm@ovirt Please enter your password:
virsh #
Dest Node After migrate --live: virsh # list Id Name State ---------------------------------------------------- 2 sl63 running
virsh #
On Thu, Jan 31, 2013 at 10:38 AM, Dead Horse < deadhorseconsulting@gmail.com> wrote:
Shu, I build oVirt Engine and vdsm from source myself. The commits I indicated are what I built from.I run the engine under FC17 and my nodes are running EL6.x respectively.
Dan, I reverted VDSM on my two test nodes to an earlier build of VDSM (commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4) VDSM after the above commit is broken due to commit: fc3a44f71d2ef202cff18d7203b9e4165b546621 however when I built and tested from master yesterday I did apply a patch I tested for ybronhei which fixed that issue.
I will build VDSM from master, today w/ the supervdsm patch and try the manual migration you indicated.
- DHC
On Thu, Jan 31, 2013 at 4:56 AM, Dan Kenigsberg <danken@redhat.com>wrote:
On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote: > On 01/30/2013 08:40 PM, Dead Horse wrote: >> The nodes are EL6.3 based. >> >> Currently installed libvirt packages: >> >> libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 >> libvirt-cim-0.6.1-3.el6.x86_64 >> libvirt-0.9.10-21.el6_3.8.x86_64 >> libvirt-python-0.9.10-21.el6_3.8.x86_64 >> libvirt-client-0.9.10-21.el6_3.8.x86_64 >> >> and qemu packages: >> qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 >> qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 >> qemu-img-0.12.1.2-2.295.el6_3.10.x86_64 >> >> Thus my presumption here given the above is that virDomainMigrateToURI2 has >> not yet been patched and/or back-ported into the EL6.x
On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote: libvirt/qemu?
>> > > virDomainMigrateToURI2 is supported since 0.9.2, but is there a > possibility the code is requesting direct migration? That might explain > the message, which is then incorrect; this was fixed in [1]. > > Martin > > [1] > http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc...
What is "direct migration" exactly, in the context of qemu-kvm?
We are using p2p migration
http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140...
OK, so that's not the issue, sorry for the confusion. I was thinking it would "somehow" get there. Direct migration doesn't exist in QEMU at all, so it seemed weird, but I can't seem to find any other reason for this failure; will keep searching, though.
In this case, Dead Horse, would you try to migrate a VM (that you do not care much about) using virsh -c qemu+tls://hostname/system migrate --p2p dsthost?
I'd like to see that the problem reproduces this way, too. More of libvirtd.log may help. You may want to disable iptables for a moment, just to eliminate a common cause of failure.
On 01/31/2013 07:07 PM, Dead Horse wrote:
Here is the content exceprt from libvirtd.log for the command: virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system
Thanks for testing this. However, this is another problem. When migrating p2p, the destination URI must be reachable from the source, not the client. That is probably what failed in first place. Would you mind trying migrating p2p (and specifying the destination URI that's reachable from the source host)? Thanks. Also, on both hosts the version are the same?
2013-01-31 18:02:53.740+0000: 2832: debug : virDomainFree:2313 : dom=0x7f4f88000c80, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) 2013-01-31 18:02:53.743+0000: 2831: debug : virDomainLookupByName:2146 : conn=0x7f4f8c001d80, name=sl63 2013-01-31 18:02:53.743+0000: 2831: debug : virDomainFree:2313 : dom=0x7f4f84002150, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb) 2013-01-31 18:02:53.747+0000: 2829: debug : virDrvSupportsFeature:1521 : conn=0x7f4f8c001d80, feature=4 2013-01-31 18:02:53.751+0000: 2826: debug : virDrvSupportsFeature:1521 : conn=0x7f4f8c001d80, feature=6 2013-01-31 18:02:53.754+0000: 2828: debug : virDomainMigratePerform3:6247 : dom=0x7f4f7c0d1430, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), xmlin=(null) cookiein=(nil), cookieinlen=0, cookieout=0x7f4f998cab30, cookieoutlen=0x7f4f998cab3c, dconnuri=qemu+ssh://192.168.1.2/system, uri=(null), flags=2, dname=(null), bandwidth=0 2013-01-31 18:02:53.755+0000: 2828: debug : qemuMigrationPerform:2821 : driver=0x7f4f8c0af820, conn=0x7f4f8c001d80, vm=0x7f4f7c0c7040, xmlin=(null), dconnuri=qemu+ssh://192.168.1.2/system<http://3.57.111.32/system>,
Funny how mail clients try to make it "easier" for us and add a link that makes sense (to them). [...]
On Thu, Jan 31, 2013 at 11:27 AM, Dead Horse <deadhorseconsulting@gmail.com>wrote:
note ignore the IP diff in the ssh host auth --> copy/paste fail ;) - DHC
On Thu, Jan 31, 2013 at 11:25 AM, Dead Horse < deadhorseconsulting@gmail.com> wrote:
Doh, brain fart VDSM is not involved here for the purposed of the needed test. Here is my initial whack at it:
Source Node:
virsh # list Id Name State ---------------------------------------------------- 1 sl63 running
virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system error: operation failed: Failed to connect to remote libvirt URI qemu+ssh://192.168.1.2/system <http://3.57.111.32/system>
virsh # migrate --live sl63 qemu+ssh://192.168.1.2/system The authenticity of host '192.168.1.2 (192.168.1.2)' can't be established. RSA key fingerprint is e5:1d:b3:e5:38:5f:e1:8b:73:26:9e:15:c8:0a:2d:ac. Are you sure you want to continue connecting (yes/no)? yes root@192.168.1.2's password: Please enter your authentication name: vdsm@ovirt Please enter your password:
virsh #
Dest Node After migrate --live: virsh # list Id Name State ---------------------------------------------------- 2 sl63 running
virsh #
On Thu, Jan 31, 2013 at 10:38 AM, Dead Horse < deadhorseconsulting@gmail.com> wrote:
Shu, I build oVirt Engine and vdsm from source myself. The commits I indicated are what I built from.I run the engine under FC17 and my nodes are running EL6.x respectively.
Dan, I reverted VDSM on my two test nodes to an earlier build of VDSM (commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4) VDSM after the above commit is broken due to commit: fc3a44f71d2ef202cff18d7203b9e4165b546621 however when I built and tested from master yesterday I did apply a patch I tested for ybronhei which fixed that issue.
I will build VDSM from master, today w/ the supervdsm patch and try the manual migration you indicated.
- DHC
On Thu, Jan 31, 2013 at 4:56 AM, Dan Kenigsberg <danken@redhat.com>wrote:
On 01/31/2013 10:25 AM, Dan Kenigsberg wrote: > On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote: >> On 01/30/2013 08:40 PM, Dead Horse wrote: >>> The nodes are EL6.3 based. >>> >>> Currently installed libvirt packages: >>> >>> libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64 >>> libvirt-cim-0.6.1-3.el6.x86_64 >>> libvirt-0.9.10-21.el6_3.8.x86_64 >>> libvirt-python-0.9.10-21.el6_3.8.x86_64 >>> libvirt-client-0.9.10-21.el6_3.8.x86_64 >>> >>> and qemu packages: >>> qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 >>> qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64 >>> qemu-img-0.12.1.2-2.295.el6_3.10.x86_64 >>> >>> Thus my presumption here given the above is that virDomainMigrateToURI2 has >>> not yet been patched and/or back-ported into the EL6.x
On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote: libvirt/qemu?
>>> >> >> virDomainMigrateToURI2 is supported since 0.9.2, but is there a >> possibility the code is requesting direct migration? That might explain >> the message, which is then incorrect; this was fixed in [1]. >> >> Martin >> >> [1] >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc... > > What is "direct migration" exactly, in the context of qemu-kvm? > > We are using p2p migration > http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140... >
OK, so that's not the issue, sorry for the confusion. I was thinking it would "somehow" get there. Direct migration doesn't exist in QEMU at all, so it seemed weird, but I can't seem to find any other reason for this failure; will keep searching, though.
In this case, Dead Horse, would you try to migrate a VM (that you do not care much about) using virsh -c qemu+tls://hostname/system migrate --p2p dsthost?
I'd like to see that the problem reproduces this way, too. More of libvirtd.log may help. You may want to disable iptables for a moment, just to eliminate a common cause of failure.
Both nodes are identical and can fully communicate with each other. Since the normal non p2p live migration works both hosts can reach each other via the connection URI. Perhaps I am missing something here? - DHC
To test further I loaded up two more identical servers with EL 6.3 and the same package versions originally indicated. The difference here is that I did not turn these into ovirt nodes. EG: installing VDSM. - All configurations were left at defaults on both servers - iptables and selinux disabled on both servers - verified full connectivty between both servers - setup ssh (/root/authorized keys) between the servers --> this turned out to be the key! Then using syntax found here: http://libvirt.org/migration.html#flowpeer2peer EG: From the source server I issued the following: virsh migrate --p2p sl63 qemu+ssh://192.168.1.2/system It fails in exactly the same way as previously indicated when the destination server does not have an ssh rsa pub ID from the source system in it's /root/.ssh/authorized_keys file. However once the ssh rsa pub ID is in place on the destination system all is well and migrations work as expected. Next I tried the same on the identical servers acting as ovirt nodes. (EG: installed with and running VDSM) The above did not do the trick, still fails the same exact way. Thus this leads me to believe something with VDSM or a configuration alteration made by it is to blame. I should note that I have SSL=disabled on my engine/nodes EG: vdc_options --> SSLEnabled = false, UseSecureConnectionWithServers = false, EnableSpiceRootCertificateValidation = false vdsm.conf (same on both nodes): [addresses] management_port = 54321 [vars] ssl = false libvirtd.conf alterations by VDSM (same on both nodes excluding host_uuid) ## beginning of configuration section by vdsm-4.9.11 <-- BTW this does not match the running build of VDSM on the system which is actually build from commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4 (vdsm-4.10.2-12.el6.x86_64, "-12" is my own internal counter) listen_addr="0.0.0.0" unix_sock_group="kvm" unix_sock_rw_perms="0770" auth_unix_rw="sasl" host_uuid="1d465e80-1e48-474f-b52d-f5aeda9f6a7a" log_outputs="1:file:/var/log/libvirtd.log" log_filters="1:libvirt 3:event 3:json 1:util 1:qemu" auth_tcp="none" listen_tcp=1 listen_tls=0 ## end of configuration section by vdsm-4.9.11 <-- Does not match running VDSM version# qemu.conf alterations by VDSM (same on both nodes) ## beginning of configuration section by vdsm-4.9.11 <-- Does not match running VDSM version# dynamic_ownership=0 spice_tls=0 save_image_format="lzop" lock_manager="sanlock" auto_dump_path="/var/log/core" ## end of configuration section by vdsm-4.9.11 <-- Does not match running VDSM version# - DHC On Fri, Feb 1, 2013 at 10:47 AM, Dead Horse <deadhorseconsulting@gmail.com>wrote:
Both nodes are identical and can fully communicate with each other. Since the normal non p2p live migration works both hosts can reach each other via the connection URI. Perhaps I am missing something here? - DHC
On 02/01/2013 09:29 PM, Dead Horse wrote:
To test further I loaded up two more identical servers with EL 6.3 and the same package versions originally indicated. The difference here is that I did not turn these into ovirt nodes. EG: installing VDSM.
- All configurations were left at defaults on both servers - iptables and selinux disabled on both servers - verified full connectivty between both servers - setup ssh (/root/authorized keys) between the servers --> this turned out to be the key!
Then using syntax found here: http://libvirt.org/migration.html#flowpeer2peer EG: From the source server I issued the following:
So your client equals to the source server, that makes us sure that the connection is made on the same network for p2p and non-p2p migration.
virsh migrate --p2p sl63 qemu+ssh://192.168.1.2/system
You're using ssh transport here, but isn't vdsm using tcp or tls? According to the config file tcp transport is enabled with no authentication whatsoever...
It fails in exactly the same way as previously indicated when the destination server does not have an ssh rsa pub ID from the source system in it's /root/.ssh/authorized_keys file. However once the ssh rsa pub ID is in place on the destination system all is well and migrations work as expected.
..., which would mean you need no ssh keys when migrating using tcp transport instead. Also during p2p migration the source libvirt daemon can't ask you for the password, but when not using p2p the client is connecting to the destination, thus being able to ask for the password and/or use different ssh keys. But it looks like none of this has anything to do with the problem as: 1) as you found out, changing vdsm versions makes the problem go away/appear and 2) IIUC the first error was "function is not supported by the connection driver: virDomainMigrateToURI2", but the second one was "error: operation failed: Failed to connect to remote libvirt URI". Since I tried finding out why the first error appeared, I probably misunderstood somewhere in the middle of this thread and am useless here. However if I can help from the libvirt POV, I'll follow up this thread and will see whether there's anything related. Good luck, Martin
On Fri, Feb 01, 2013 at 11:44:08PM +0100, Martin Kletzander wrote:
On 02/01/2013 09:29 PM, Dead Horse wrote:
To test further I loaded up two more identical servers with EL 6.3 and the same package versions originally indicated. The difference here is that I did not turn these into ovirt nodes. EG: installing VDSM.
- All configurations were left at defaults on both servers - iptables and selinux disabled on both servers - verified full connectivty between both servers - setup ssh (/root/authorized keys) between the servers --> this turned out to be the key!
Then using syntax found here: http://libvirt.org/migration.html#flowpeer2peer EG: From the source server I issued the following:
So your client equals to the source server, that makes us sure that the connection is made on the same network for p2p and non-p2p migration.
virsh migrate --p2p sl63 qemu+ssh://192.168.1.2/system
You're using ssh transport here, but isn't vdsm using tcp or tls?
It is!
According to the config file tcp transport is enabled with no authentication whatsoever...
It fails in exactly the same way as previously indicated when the destination server does not have an ssh rsa pub ID from the source system in it's /root/.ssh/authorized_keys file. However once the ssh rsa pub ID is in place on the destination system all is well and migrations work as expected.
..., which would mean you need no ssh keys when migrating using tcp transport instead.
Also during p2p migration the source libvirt daemon can't ask you for the password, but when not using p2p the client is connecting to the destination, thus being able to ask for the password and/or use different ssh keys.
But it looks like none of this has anything to do with the problem as:
1) as you found out, changing vdsm versions makes the problem go away/appear and
I've missed this point. Which version of vdsm makes it go away?
2) IIUC the first error was "function is not supported by the connection driver: virDomainMigrateToURI2", but the second one was "error: operation failed: Failed to connect to remote libvirt URI".
Since I tried finding out why the first error appeared, I probably misunderstood somewhere in the middle of this thread and am useless here. However if I can help from the libvirt POV, I'll follow up this thread and will see whether there's anything related.
On 02/03/2013 08:40 AM, Dan Kenigsberg wrote:
On Fri, Feb 01, 2013 at 11:44:08PM +0100, Martin Kletzander wrote:
On 02/01/2013 09:29 PM, Dead Horse wrote:
To test further I loaded up two more identical servers with EL 6.3 and the same package versions originally indicated. The difference here is that I did not turn these into ovirt nodes. EG: installing VDSM.
- All configurations were left at defaults on both servers - iptables and selinux disabled on both servers - verified full connectivty between both servers - setup ssh (/root/authorized keys) between the servers --> this turned out to be the key!
Then using syntax found here: http://libvirt.org/migration.html#flowpeer2peer EG: From the source server I issued the following:
So your client equals to the source server, that makes us sure that the connection is made on the same network for p2p and non-p2p migration.
virsh migrate --p2p sl63 qemu+ssh://192.168.1.2/system
You're using ssh transport here, but isn't vdsm using tcp or tls?
It is!
So then testing it with '+ssh' does not help much. But at least we know the addresses are reachable.
According to the config file tcp transport is enabled with no authentication whatsoever...
It fails in exactly the same way as previously indicated when the destination server does not have an ssh rsa pub ID from the source system in it's /root/.ssh/authorized_keys file. However once the ssh rsa pub ID is in place on the destination system all is well and migrations work as expected.
..., which would mean you need no ssh keys when migrating using tcp transport instead.
Also during p2p migration the source libvirt daemon can't ask you for the password, but when not using p2p the client is connecting to the destination, thus being able to ask for the password and/or use different ssh keys.
But it looks like none of this has anything to do with the problem as:
1) as you found out, changing vdsm versions makes the problem go away/appear and
I've missed this point. Which version of vdsm makes it go away?
Sorry, I've got it stuck in my head that part of the thread was about it, but when going through the mail now it makes less sense than before. I probably understood that from [1] and maybe some other sentence that mixed in my head, but was related to the ssh migration. Sorry for that, Martin [1] http://www.mail-archive.com/users@ovirt.org/msg06105.html
VDSM built from commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4 works Current VDSM built from master does not work. I could try spending some time trying to bisect and find out where the breakage occurred I suppose. - DHC On Sun, Feb 3, 2013 at 10:13 AM, Martin Kletzander <mkletzan@redhat.com>wrote:
On Fri, Feb 01, 2013 at 11:44:08PM +0100, Martin Kletzander wrote:
On 02/01/2013 09:29 PM, Dead Horse wrote:
To test further I loaded up two more identical servers with EL 6.3 and
same package versions originally indicated. The difference here is
On 02/03/2013 08:40 AM, Dan Kenigsberg wrote: the that I
did not turn these into ovirt nodes. EG: installing VDSM.
- All configurations were left at defaults on both servers - iptables and selinux disabled on both servers - verified full connectivty between both servers - setup ssh (/root/authorized keys) between the servers --> this turned out to be the key!
Then using syntax found here: http://libvirt.org/migration.html#flowpeer2peer EG: From the source server I issued the following:
So your client equals to the source server, that makes us sure that the connection is made on the same network for p2p and non-p2p migration.
virsh migrate --p2p sl63 qemu+ssh://192.168.1.2/system
You're using ssh transport here, but isn't vdsm using tcp or tls?
It is!
So then testing it with '+ssh' does not help much. But at least we know the addresses are reachable.
According to the config file tcp transport is enabled with no authentication whatsoever...
It fails in exactly the same way as previously indicated when the destination server does not have an ssh rsa pub ID from the source system in it's /root/.ssh/authorized_keys file. However once the ssh rsa pub ID is in place on the destination system all is well and migrations work as expected.
..., which would mean you need no ssh keys when migrating using tcp transport instead.
Also during p2p migration the source libvirt daemon can't ask you for the password, but when not using p2p the client is connecting to the destination, thus being able to ask for the password and/or use different ssh keys.
But it looks like none of this has anything to do with the problem as:
1) as you found out, changing vdsm versions makes the problem go away/appear and
I've missed this point. Which version of vdsm makes it go away?
Sorry, I've got it stuck in my head that part of the thread was about it, but when going through the mail now it makes less sense than before. I probably understood that from [1] and maybe some other sentence that mixed in my head, but was related to the ssh migration.
Sorry for that, Martin
[1] http://www.mail-archive.com/users@ovirt.org/msg06105.html
On Mon, Feb 04, 2013 at 10:38:16AM -0600, Dead Horse wrote:
VDSM built from commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4 works Current VDSM built from master does not work.
I could try spending some time trying to bisect and find out where the breakage occurred I suppose.
Would you be kind to find the time to help us here? Clearly, the commit on top of c343e1833f7b6e5428dd90f14f7807dca1baa0b4 introduces nasty supervdsm regressions; it has to be reverted for any meaningful testing. However I do not see how it can be related to the problem at hand. Would you at least try out the ovirt-3.2 branch (where the infamous "zombie reaper" commit is reverted)? Dan.
Dan, Building and testing it now. - DHC On Tue, Feb 5, 2013 at 2:39 AM, Dan Kenigsberg <danken@redhat.com> wrote:
On Mon, Feb 04, 2013 at 10:38:16AM -0600, Dead Horse wrote:
VDSM built from commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4 works Current VDSM built from master does not work.
I could try spending some time trying to bisect and find out where the breakage occurred I suppose.
Would you be kind to find the time to help us here? Clearly, the commit on top of c343e1833f7b6e5428dd90f14f7807dca1baa0b4 introduces nasty supervdsm regressions; it has to be reverted for any meaningful testing. However I do not see how it can be related to the problem at hand.
Would you at least try out the ovirt-3.2 branch (where the infamous "zombie reaper" commit is reverted)?
Dan.
Confirmed ovirt-3.2 branch of vdsm does work with migrations. So there is a difference between it and master as pertains to the migration logic. - DHC On Tue, Feb 5, 2013 at 2:26 PM, Dead Horse <deadhorseconsulting@gmail.com>wrote:
Dan, Building and testing it now. - DHC
On Tue, Feb 5, 2013 at 2:39 AM, Dan Kenigsberg <danken@redhat.com> wrote:
On Mon, Feb 04, 2013 at 10:38:16AM -0600, Dead Horse wrote:
VDSM built from commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4 works Current VDSM built from master does not work.
I could try spending some time trying to bisect and find out where the breakage occurred I suppose.
Would you be kind to find the time to help us here? Clearly, the commit on top of c343e1833f7b6e5428dd90f14f7807dca1baa0b4 introduces nasty supervdsm regressions; it has to be reverted for any meaningful testing. However I do not see how it can be related to the problem at hand.
Would you at least try out the ovirt-3.2 branch (where the infamous "zombie reaper" commit is reverted)?
Dan.
This is a multi-part message in MIME format. --------------010408040801010404090600 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 02/05/2013 10:57 PM, Dead Horse wrote:
Confirmed ovirt-3.2 branch of vdsm does work with migrations. So there is a difference between it and master as pertains to the migration logic. - DHC Hi,
could you please retry it with the current master? We have fixed the issue which caused this regression. :-) Thank you :)
On Tue, Feb 5, 2013 at 2:26 PM, Dead Horse <deadhorseconsulting@gmail.com <mailto:deadhorseconsulting@gmail.com>> wrote:
Dan, Building and testing it now. - DHC
On Tue, Feb 5, 2013 at 2:39 AM, Dan Kenigsberg <danken@redhat.com <mailto:danken@redhat.com>> wrote:
On Mon, Feb 04, 2013 at 10:38:16AM -0600, Dead Horse wrote: > VDSM built from commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4 works > Current VDSM built from master does not work. > > I could try spending some time trying to bisect and find out where the > breakage occurred I suppose.
Would you be kind to find the time to help us here? Clearly, the commit on top of c343e1833f7b6e5428dd90f14f7807dca1baa0b4 introduces nasty supervdsm regressions; it has to be reverted for any meaningful testing. However I do not see how it can be related to the problem at hand.
Would you at least try out the ovirt-3.2 branch (where the infamous "zombie reaper" commit is reverted)?
Dan.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com --------------010408040801010404090600 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 02/05/2013 10:57 PM, Dead Horse wrote:<br> </div> <blockquote cite="mid:CAEWPe=oB+2btdvRNMJNeK53m-EqcgDK72NfxgzgVsyqN0RAYDw@mail.gmail.com" type="cite"> <div dir="ltr"> <div> <div>Confirmed ovirt-3.2 branch of vdsm does work with migrations.<br> </div> So there is a difference between it and master as pertains to the migration logic.<br> </div> - DHC<br> </div> </blockquote> Hi,<br> <br> could you please retry it with the current master? We have fixed the issue which caused this regression. :-)<br> Thank you :)<br> <br> <blockquote cite="mid:CAEWPe=oB+2btdvRNMJNeK53m-EqcgDK72NfxgzgVsyqN0RAYDw@mail.gmail.com" type="cite"> <div class="gmail_extra"> <br> <br> <div class="gmail_quote">On Tue, Feb 5, 2013 at 2:26 PM, Dead Horse <span dir="ltr"><<a moz-do-not-send="true" href="mailto:deadhorseconsulting@gmail.com" target="_blank">deadhorseconsulting@gmail.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div> <div>Dan,<br> </div> Building and testing it now.<br> </div> - DHC<br> </div> <div class="HOEnZb"> <div class="h5"> <div class="gmail_extra"><br> <br> <div class="gmail_quote">On Tue, Feb 5, 2013 at 2:39 AM, Dan Kenigsberg <span dir="ltr"><<a moz-do-not-send="true" href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div>On Mon, Feb 04, 2013 at 10:38:16AM -0600, Dead Horse wrote:<br> > VDSM built from commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4 works<br> > Current VDSM built from master does not work.<br> ><br> > I could try spending some time trying to bisect and find out where the<br> > breakage occurred I suppose.<br> <br> </div> Would you be kind to find the time to help us here? Clearly, the commit<br> on top of c343e1833f7b6e5428dd90f14f7807dca1baa0b4 introduces nasty<br> supervdsm regressions; it has to be reverted for any meaningful testing.<br> However I do not see how it can be related to the problem at hand.<br> <br> Would you at least try out the ovirt-3.2 branch (where the infamous<br> "zombie reaper" commit is reverted)?<br> <span><font color="#888888"><br> Dan.<br> </font></span></blockquote> </div> <br> </div> </div> </div> </blockquote> </div> <br> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre class="moz-signature" cols="72">-- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com</pre> </body> </html> --------------010408040801010404090600--
Current master does resolve the issue. However in order to test it the zombie reaper patch from the ovirt-3.2 branch must be applied to avoid that issue. - DHC On Fri, Feb 8, 2013 at 2:08 AM, Vinzenz Feenstra <vfeenstr@redhat.com>wrote:
On 02/05/2013 10:57 PM, Dead Horse wrote:
Confirmed ovirt-3.2 branch of vdsm does work with migrations. So there is a difference between it and master as pertains to the migration logic. - DHC
Hi,
could you please retry it with the current master? We have fixed the issue which caused this regression. :-) Thank you :)
On Tue, Feb 5, 2013 at 2:26 PM, Dead Horse <deadhorseconsulting@gmail.com>wrote:
Dan, Building and testing it now. - DHC
On Tue, Feb 5, 2013 at 2:39 AM, Dan Kenigsberg <danken@redhat.com> wrote:
On Mon, Feb 04, 2013 at 10:38:16AM -0600, Dead Horse wrote:
VDSM built from commit: c343e1833f7b6e5428dd90f14f7807dca1baa0b4 works Current VDSM built from master does not work.
I could try spending some time trying to bisect and find out where the breakage occurred I suppose.
Would you be kind to find the time to help us here? Clearly, the commit on top of c343e1833f7b6e5428dd90f14f7807dca1baa0b4 introduces nasty supervdsm regressions; it has to be reverted for any meaningful testing. However I do not see how it can be related to the problem at hand.
Would you at least try out the ovirt-3.2 branch (where the infamous "zombie reaper" commit is reverted)?
Dan.
_______________________________________________ Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
-- Regards,
Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo
Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Fri, Feb 08, 2013 at 01:40:59PM -0600, Dead Horse wrote:
Current master does resolve the issue. However in order to test it the zombie reaper patch from the ovirt-3.2 branch must be applied to avoid that issue. - DHC
I tend to agree that keeping vdsm's master branch broken for too long is not very friendly. If the issue is not fixed properly in the coming few days, I would have to resort to a plain revert http://gerrit.ovirt.org/#/c/11492/ (that reexposes process leak).
participants (5)
-
Dan Kenigsberg -
Dead Horse -
Martin Kletzander -
Shu Ming -
Vinzenz Feenstra