Problem migration VM
Hi list! Sorry fort the previous E-Mail... problem on my Outlook... :( Here again... After a "war-week" I finally got a systemd-script to put the host in "maintenance" when a shutdown will started. Now the problem is, that the automatically migration of the VM does NOT work... I see in the Web console the host will "Preparing for maintenance" and the VM will start the migration, then the host is in "maintenance" and a couple of seconds later the VM will be killed on the other host... In the Log of the engine I see: 2015-09-23 11:14:17,165 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [683624fe] Correlation ID: 52938d4d, Job ID: c0efe5b9-0bc3-4c81-9ee7-63ddf90a6afc, Call Stack: null, Custom Event ID: -1, Message: Mig ration failed while Host is in 'preparing for maintenance' state. Consider manual intervention: stopping/migrating Vms as Host's state will not turn to maintenance while VMs are still running on it.(VM: TestVM, Source: vmhost06, Destination: vmhost03). 2015-09-23 11:14:17,165 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-8-thread-30) [683624fe] Lock freed to object EngineLock [exclusiveLocks= key: aabf6e76-8387-4441-a328-6a7dc32e2b4d value: VM , sharedLocks= ] (see http://pastebin.com/3Ca8W3vE) On the host I see these two errors: libvirtEventLoop::ERROR::2015-09-23 11:14:14,690::task::866::Storage.TaskManager.Task::(_setError) Task=`2670e82a-c9c7-4da6-b6f6-cff7bce25da1`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3209, in inappropriateDevices fails = supervdsm.getProxy().rmAppropriateRules(thiefId) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in rmAppropriateRules File "/usr/lib64/python2.7/multiprocessing/managers.py", line 755, in _callmethod self._connect() File "/usr/lib64/python2.7/multiprocessing/managers.py", line 742, in _connect conn = self._Client(self._token.address, authkey=self._authkey) File "/usr/lib64/python2.7/multiprocessing/connection.py", line 173, in Client c = SocketClient(address) File "/usr/lib64/python2.7/multiprocessing/connection.py", line 308, in SocketClient s.connect(address) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) error: [Errno 2] No such file or directory libvirtEventLoop::ERROR::2015-09-23 11:14:14,696::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 2] No such file or directory Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 103, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1179, in prepare raise self.error error: [Errno 2] No such file or directory libvirtEventLoop::DEBUG::2015-09-23 11:14:14,697::vm::2799::vm.Vm::(setDownStatus) vmId=`aabf6e76-8387-4441-a328-6a7dc32e2b4d`::Changed state to Down: User shut down from within the guest (code=7) libvirtEventLoop::DEBUG::2015-09-23 11:14:14,698::sampling::425::vm.Vm::(stop) vmId=`aabf6e76-8387-4441-a328-6a7dc32e2b4d`::Stop statistics collection Thread-891::ERROR::2015-09-23 11:14:14,704::migration::161::vm.Vm::(_recover) vmId=`aabf6e76-8387-4441-a328-6a7dc32e2b4d`::'NoneType' object has no attribute 'XMLDesc' Thread-891::WARNING::2015-09-23 11:14:14,712::vm::1966::vm.Vm::(_set_lastStatus) vmId=`aabf6e76-8387-4441-a328-6a7dc32e2b4d`::trying to set state to Up when already Down Thread-891::ERROR::2015-09-23 11:14:14,712::migration::260::vm.Vm::(run) vmId=`aabf6e76-8387-4441-a328-6a7dc32e2b4d`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 231, in run self._setupRemoteMachineParams() File "/usr/share/vdsm/virt/migration.py", line 132, in _setupRemoteMachineParams self._machineParams['_srcDomXML'] = self._vm._dom.XMLDesc(0) AttributeError: 'NoneType' object has no attribute 'XMLDesc' Can someone help me finding the problem? Thanks Mit freundlichen Grüßen Luca Bertoncello -- Besuchen Sie unsere Webauftritte: www.queo.biz Agentur für Markenführung und Kommunikation www.queoflow.com IT-Consulting und Individualsoftwareentwicklung Luca Bertoncello Administrator Telefon: +49 351 21 30 38 0 Fax: +49 351 21 30 38 99 E-Mail: l.bertoncello@queo-group.com queo GmbH Tharandter Str. 13 01159 Dresden Sitz der Gesellschaft: Dresden Handelsregistereintrag: Amtsgericht Dresden HRB 22352 Geschäftsführer: Rüdiger Henke, André Pinkert USt-IdNr.: DE234220077
On 23/09/15 10:30, Luca Bertoncello wrote:
Hi list!
Sorry fort the previous E-Mail... problem on my Outlook... :( Here again...
After a "war-week" I finally got a systemd-script to put the host in "maintenance" when a shutdown will started. Now the problem is, that the automatically migration of the VM does NOT work...
I see in the Web console the host will "Preparing for maintenance" and the VM will start the migration, then the host is in "maintenance" and a couple of seconds later the VM will be killed on the other host... Did the host you are running the script on shut down before the migration completed?
If you put the host in maintenance from the GUI, does it successfully migrate off all VMs? Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. "Transact" is operated by Integrated Financial Arrangements plc. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). .
Hi Alex
Did the host you are running the script on shut down before the migration completed?
Apparently yes...
If you put the host in maintenance from the GUI, does it successfully migrate off all VMs?
Yes, this happens! Now, I think, the problem is that the system first unmount /run and then call my script, so that libvirt has no more possibility to successfully migrate the VMs... Can someone suggest me a way to call my script as FIRST script on shutdown/reboot and to block the shutdown/reboot until my script complete? This will solve the problem... Of course, I can try with a wrapper for /sbin/shutdown and /sbin/reboot, but this is not a nice solution... Thanks Mit freundlichen Grüßen Luca Bertoncello -- Besuchen Sie unsere Webauftritte: www.queo.biz Agentur für Markenführung und Kommunikation www.queoflow.com IT-Consulting und Individualsoftwareentwicklung Luca Bertoncello Administrator Telefon: +49 351 21 30 38 0 Fax: +49 351 21 30 38 99 E-Mail: l.bertoncello@queo-group.com queo GmbH Tharandter Str. 13 01159 Dresden Sitz der Gesellschaft: Dresden Handelsregistereintrag: Amtsgericht Dresden HRB 22352 Geschäftsführer: Rüdiger Henke, André Pinkert USt-IdNr.: DE234220077
On 23/09/15 13:54, Luca Bertoncello wrote:
Hi Alex
Did the host you are running the script on shut down before the migration completed? Apparently yes...
Thought so. The migration then cannot continue, obviously.
If you put the host in maintenance from the GUI, does it successfully migrate off all VMs? Yes, this happens!
Now, I think, the problem is that the system first unmount /run and then call my script, so that libvirt has no more possibility to successfully migrate the VMs...
Can someone suggest me a way to call my script as FIRST script on shutdown/reboot and to block the shutdown/reboot until my script complete? This will solve the problem...
Add your systemd script as a "requires" entry in the systemd script responsible for shutting down the system?
Of course, I can try with a wrapper for /sbin/shutdown and /sbin/reboot, but this is not a nice solution...
Why don't you manage this from another machine, not from the hosts? Just have a script call the API to initiate maintenance, wait for the migration to complete, then call the API to shut down the host? Or is it really too hard to do this from the GUI? I don't understand why you have such a hard requirement to be able to do this from the hosts - the whole point of Ovirt is that you don't have to manage your hosts on an individual basis! Alex
Thanks
Mit freundlichen Grüßen
Luca Bertoncello
-- Besuchen Sie unsere Webauftritte:
www.queo.biz Agentur für Markenführung und Kommunikation www.queoflow.com IT-Consulting und Individualsoftwareentwicklung
Luca Bertoncello Administrator Telefon: +49 351 21 30 38 0 Fax: +49 351 21 30 38 99 E-Mail: l.bertoncello@queo-group.com
queo GmbH Tharandter Str. 13 01159 Dresden Sitz der Gesellschaft: Dresden Handelsregistereintrag: Amtsgericht Dresden HRB 22352 Geschäftsführer: Rüdiger Henke, André Pinkert USt-IdNr.: DE234220077 _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. "Transact" is operated by Integrated Financial Arrangements plc. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). .
Hi Alex
Thought so. The migration then cannot continue, obviously.
Obviously... :(
Can someone suggest me a way to call my script as FIRST script on shutdown/reboot and to block the shutdown/reboot until my script complete? This will solve the problem...
Add your systemd script as a "requires" entry in the systemd script responsible for shutting down the system?
What do you mean? Could you please explain, maybe with an example?
Of course, I can try with a wrapper for /sbin/shutdown and /sbin/reboot, but this is not a nice solution...
Why don't you manage this from another machine, not from the hosts? Just have a script call the API to initiate maintenance, wait for the migration to complete, then call the API to shut down the host?
Well, the problem is, that the host MUST be able to shutdown and migrate the VMs if for example we have a black-out... The UPS send a signal and NUT start the shutdown. In this case, the VMs have to migrate.
Or is it really too hard to do this from the GUI? I don't understand why you have such a hard requirement to be able to do this from the hosts - the whole point of Ovirt is that you don't have to manage your hosts on an individual basis!
Well, the problem is just one: if someone other Admin has to perform some works on the host that require a reboot, and he MUST log into a GUI to put the host in maintenance before the shutdown, we have a higher fail-possibility, if he forgot that. And, of course, automatically shutdown cannot log into the GUI... So, the best solution is, that the host itself, BEFORE the shutdown begins, send oVirt a signal and let oVirt put the host in Maintenance, migrating all VMs, and THEN, proceed with the shutdown... I think, it MUST be possible, but I can't find how... Thanks Mit freundlichen Grüßen Luca Bertoncello -- Besuchen Sie unsere Webauftritte: www.queo.biz Agentur für Markenführung und Kommunikation www.queoflow.com IT-Consulting und Individualsoftwareentwicklung Luca Bertoncello Administrator Telefon: +49 351 21 30 38 0 Fax: +49 351 21 30 38 99 E-Mail: l.bertoncello@queo-group.com queo GmbH Tharandter Str. 13 01159 Dresden Sitz der Gesellschaft: Dresden Handelsregistereintrag: Amtsgericht Dresden HRB 22352 Geschäftsführer: Rüdiger Henke, André Pinkert USt-IdNr.: DE234220077
On 23/09/15 14:11, Luca Bertoncello wrote:
Hi Alex
Thought so. The migration then cannot continue, obviously. Obviously... :(
Can someone suggest me a way to call my script as FIRST script on shutdown/reboot and to block the shutdown/reboot until my script complete? This will solve the problem... Add your systemd script as a "requires" entry in the systemd script responsible for shutting down the system? What do you mean? Could you please explain, maybe with an example?
I've not done anything with systemd myself but it should be entirely possible. Ask systemd people.
Of course, I can try with a wrapper for /sbin/shutdown and /sbin/reboot, but this is not a nice solution...
Why don't you manage this from another machine, not from the hosts? Just have a script call the API to initiate maintenance, wait for the migration to complete, then call the API to shut down the host?
OK, change the shutdown script on the host that NUT calls (ie *not* the systemd stuff) to do what I said above. That would work. You will have to be careful with your timing to ensure that migration can finish before your UPS runs out of juice though. NUT normally only starts the shutdown process on LOWBATT from the UPS.
Or is it really too hard to do this from the GUI? I don't understand why you have such a hard requirement to be able to do this from the hosts - the whole point of Ovirt is that you don't have to manage your hosts on an individual basis! Well, the problem is just one: if someone other Admin has to perform some works on the host that require a reboot, and he MUST log into a GUI to put the host in maintenance before the shutdown, we have a higher fail-possibility, if he forgot that. And, of course, automatically shutdown cannot log into the GUI...
No, but it can have access to the API. You'd have to mess around with the systemd scripts to do this bit. As above, I think you'd be better off asking the systemd people than this list - as this is now getting quite offtopic for here. Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. "Transact" is operated by Integrated Financial Arrangements plc. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). .
participants (2)
-
Alex Crow -
Luca Bertoncello