[ovirt-users] Re: Can't manually migrate VM's (4.3.0)

Just upgraded all the hosts from 4.2.8 to 4.3.2 - everything is running fine as of now apart from this is issue - while manually migrating VMS to other hosts having enough capacity I am getting the same error - "Could not fetch data needed for VM migrate operation" over UI I can see the same VDSM errors on each node, ========================================== ERROR Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' ========================================= Please let me know if this relevant to the same issue? On Fri, Mar 1, 2019 at 9:40 PM Greg Sheremeta <gshereme(a)redhat.com&gt; wrote: > You are correct -- it's new. We just discovered it about 2 weeks ago. > That's how I knew to point you to it :D > > Best wishes, > Greg > > On Fri, Mar 1, 2019 at 11:03 AM Ron Jerome <ronjero(a)gmail.com&gt; wrote: > >> Bingo!! That fixed the issue. >> >> Thanks Greg. >> >> Just one final question on this... Is that parameter new? I could have >> sworn that I just cut and pasted that section of the docs (modifying as >> appropriate) into my squid.conf file when I set it up. >> >> Ron. >> >> On Fri, 1 Mar 2019 at 10:02, Greg Sheremeta <gshereme(a)redhat.com&gt; wrote: >> >>> >>> On Fri, Mar 1, 2019 at 9:47 AM Ron Jerome <ronjero(a)gmail.com&gt; wrote: >>> >>>> Thanks Michal, >>>> >>>> I think we are onto something here. That request is getting a 401 >>>> unauthorized response... >>>> >>>> ssl_access_log:10.10.10.41 - - [01/Mar/2019:09:26:46 -0500] "GET >>>> /ovirt-engine/api/vms/?search=id=dc0a6167-3c36-48e4-9cca-d69303037859 >>>> HTTP/1.1" 401 71 >>>> >>>> I guess it should be noted here that I'm accessing the engine through >>>> a squid proxy on one of the hosts. I just tested a direct connection to >>>> the engine (without going through the proxy) and it works, so the next >>>> question is how to fix the proxy issue? Could this be an SSL certificate >>>> issue? >>>> >>> >>> Oh, squid :) >>> Please verify that you have "login=PASSTHRU" set. >>> Like in #14 here: >>> https://www.ovirt.org/documentation/admin-guide/chap-Proxies.html#install... >>> >>> >>>> Ron. >>>> >>>> On Fri, 1 Mar 2019 at 04:50, Michal Skrivanek <mskrivan(a)redhat.com&gt; >>>> wrote: >>>> >>>>> >>>>> >>>>> On 1 Mar 2019, at 02:34, Ron Jerome <ronjero(a)gmail.com&gt; wrote: >>>>> >>>>> Here is the JS error that is being generated when I push the >>>>> "Migrate" button... >>>>> >>>>> DataProvider failed to fetch data SyntaxError: JSON.parse: unexpected >>>>>> character at line 1 column 1 of the JSON data DataProvider.js:35:8 >>>>>> >>>>>> value/< >>>>>> DataProvider.js:35:8 >>>>>> A/</< >>>>>> >>>>>> es6.promise.js:75 >>>>>> >>>>>> A/< >>>>>> es6.promise.js:60 >>>>>> l >>>>>> _ >>>>>> microtask.js:18 >>>>>> >>>>> >>>>> >>>>> Hi, >>>>> without a reproduced you’d need to provide a bit more info. Is there >>>>> anything else in the console? >>>>> It would be best to understand which requests returned invalid data, >>>>> for that you can enable the network monitoring and grab the request and >>>>> response perhaps? We need to see if it’s really the migration request or >>>>> anything else, and what was the actual response >>>>> >>>>> Thanks, >>>>> michal >>>>> >>>>> Ron. >>>>> >>>>> On Wed, 27 Feb 2019 at 17:26, Greg Sheremeta <gshereme(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> Hi Ron, >>>>>> >>>>>> Please attach your browser console log and engine.log snippet when >>>>>> you have the problem. >>>>>> If you could dive into the console and grab the actual REST API >>>>>> response, that would be great. >>>>>> The request will be something like >>>>>> <engine>/api/hosts?migration_target_of=... >>>>>> >>>>>> Greg >>>>>> >>>>>> On Tue, Feb 26, 2019 at 6:32 PM Greg Sheremeta <gshereme(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> It sounds like a bug. I'll talk with Michal about reverting this >>>>>>> dialog to the 4.2 version. >>>>>>> >>>>>>> Greg >>>>>>> >>>>>>> On Tue, Feb 26, 2019 at 1:47 PM Ron Jerome <ronjero(a)gmail.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Sharon, >>>>>>>> This happens with all the VM's, regardless of uptime. I've never >>>>>>>> tried to migrate a VM if it's not completely up. >>>>>>>> >>>>>>>> When I select a VM and push the "Migrate" button, I never get to >>>>>>>> the "Migrate" dialog box, I just presents the error show in the attached >>>>>>>> image. >>>>>>>> >>>>>>>> I know for a fact that manual migration did work on this cluster, >>>>>>>> unfortunately, I don't know if it ever worked in 4.3.0 or if I was still at >>>>>>>> 4.2.8 when I last used it. >>>>>>>> >>>>>>>> Ron. >>>>>>>> >>>>>>>> On Tue, 26 Feb 2019 at 11:59, Sharon Gratch <sgratch(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Ron, >>>>>>>>> >>>>>>>>> What is the VM state when you try to manually migrate it? Is the >>>>>>>>> VM in the beginning of the "powering up" state? >>>>>>>>> If so then please wait 1-2 seconds and then try to migrate again >>>>>>>>> and see if it is reproduced. >>>>>>>>> You can't migrate a VM on "Wait for Launch" state and once the VM >>>>>>>>> enters the "powering up" state then it sometimes takes time for UI to be >>>>>>>>> refreshed with state and data. It was reproduced to me too. >>>>>>>>> >>>>>>>>> Greg, does it sound reasonable? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Sharon >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Can you please send a screenshot of the "Migrate VM(s)" dialog >>>>>>>>> with th >>>>>>>>> >>>>>>>>> On Tue, Feb 26, 2019 at 5:33 PM Ron Jerome <ronjero(a)gmail.com&gt; >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I've toggled all the hosts into and out of maintenance, and VM's >>>>>>>>>> migrate off of each as expected, but I still can't manually initiate a VM >>>>>>>>>> migration from the UI. Do you have any hints as to where to look for error >>>>>>>>>> messages? >>>>>>>>>> >>>>>>>>>> Thanks in advance, >>>>>>>>>> >>>>>>>>>> Ron. >>>>>>>>>> >>>>>>>>>> On Mon, 25 Feb 2019 at 19:56, Ron Jerome <ronjero(a)gmail.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It's a 3 node cluster, each node has 84G RAM, and there are >>>>>>>>>>> only two two other VM's running, so there should be plenty of capacity. >>>>>>>>>>> >>>>>>>>>>> Automatic migration works, if I put a host into Maintenance, >>>>>>>>>>> the VM's will migrate. >>>>>>>>>>> >>>>>>>>>>> Ron >>>>>>>>>>> >>>>>>>>>>> On Mon, Feb 25, 2019, 6:46 PM Greg Sheremeta, < >>>>>>>>>>> gshereme(a)redhat.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>>> Turns out it's a bad error message. It just means there are no >>>>>>>>>>>> hosts available to migrate to. >>>>>>>>>>>> >>>>>>>>>>>> Do you have other hosts up with capacity? >>>>>>>>>>>> >>>>>>>>>>>> Greg >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Feb 25, 2019 at 3:01 PM Ron Jerome <ronjero(a)gmail.com&gt; >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I've been running 4.3.0 for a few weeks now and just >>>>>>>>>>>>> discovered that I can't manually migrate VM's from the UI. I get an error >>>>>>>>>>>>> message saying: "Could not fetch data needed for VM migrate >>>>>>>>>>>>> operation" >>>>>>>>>>>>> >>>>>>>>>>>>> Sounds like >>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1670701 >>>>>>>>>>>>> >>>>>>>>>>>>> Ron. >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>>> List Archives: >>>>>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/5OBUNZHUPVE... >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> GREG SHEREMETA >>>>>>>>>>>> >>>>>>>>>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>>>>>>>>>>> Red Hat NA >>>>>>>>>>>> >>>>>>>>>>>> <https://www.redhat.com/> >>>>>>>>>>>> >>>>>>>>>>>> gshereme(a)redhat.com IRC: gshereme >>>>>>>>>>>> <https://red.ht/sig> >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: >>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/VVSXK3WO4AA... >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> GREG SHEREMETA >>>>>>> >>>>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>>>>>> Red Hat NA >>>>>>> >>>>>>> <https://www.redhat.com/> >>>>>>> >>>>>>> gshereme(a)redhat.com IRC: gshereme >>>>>>> <https://red.ht/sig> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> GREG SHEREMETA >>>>>> >>>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>>>>> Red Hat NA >>>>>> >>>>>> <https://www.redhat.com/> >>>>>> >>>>>> gshereme(a)redhat.com IRC: gshereme >>>>>> <https://red.ht/sig> >>>>>> >>>>> >>>>> >>> >>> -- >>> >>> GREG SHEREMETA >>> >>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>> >>> Red Hat NA >>> >>> <https://www.redhat.com/> >>> >>> gshereme(a)redhat.com IRC: gshereme >>> <https://red.ht/sig> >>> >> > > -- > > GREG SHEREMETA > > SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX > > Red Hat NA > > <https://www.redhat.com/> > > gshereme(a)redhat.com IRC: gshereme > <https://red.ht/sig> > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5EC3LX3MVXE... > -- ABHISHEK SAHNI Mob : +91-990-701-5143