Auto-SOLVED, but read anyway : Invalid status on Data Center. Setting status to Non Responsive.
Hi, On our second oVirt setup in 3.4.0-1.el6 (that was running fine), I did a yum upgrade on the engine (...sigh...). Then rebooted the engine. This machine is hosting the NFS export domain. Though the VM are still running, the storage domain is in invalid status. You'll find below the engine.log. At first sight, I thought it was the same issue as : http://lists.ovirt.org/pipermail/users/2014-March/022161.html because it looked very similar. But the NFS export domain connection seemed OK (tested). I did try every trick I could thought of, restarting, checking anything... Our cluster stayed in a broken state. On second sight, I saw that when rebooting the engine, then NFS export domain was not mounted correctly (I wrote a static /dev/sd-something in fstab, and the iscsi manager changed the letter. Next time, I'll use LVM or a label). So the NFS served was void/empty/black hole. I just realized all the above, and spent my afternoon in cold sweat. Correcting the NFS mounting and restarting the engine did the trick. What still disturbs me is that the unavailability of the NFS export domain should NOT be a reason for the MASTER storage domain to break! Following the URL above and the BZ opened by the user (https://bugzilla.redhat.com/show_bug.cgi?id=1072900), I see this has been corrected in 3.4.1. What gives a perfectly connected NFS export domain, but empty? PS : I see no 3.4.1 update on CentOS repo. Regards, -------------------------- The engine log : 2014-05-09 14:40:37,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] spmStart polling started: taskId = 6d612398-fdad-49f2-9874-5f32a9bf87e2 │ 20│2014-05-09 14:40:40,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] Failed in HSMGetTaskStatusVDS method │ 20│2014-05-09 14:40:40,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] spmStart polling ended: taskId = 6d612398-fdad-49f2-9874-5f32a9bf87e2 task status = finished │ 20│2014-05-09 14:40:40,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 │ 20│2014-05-09 14:40:40,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] spmStart polling ended, spm status: Free │ 20│2014-05-09 14:40:40,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] START, HSMClearTaskVDSCommand(HostName = serv-vm-adm17, HostId = 049943eb-2bcc-4167-a780-7ef76a1f95e9, taskId=6d612398-fdad-49f2-9874-5f32a9bf87e2), log id: 5cfdc8ce │ 20│2014-05-09 14:40:40,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] FINISH, HSMClearTaskVDSCommand, log id: 5cfdc8ce │ 20│2014-05-09 14:40:40,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@39471ba9, log id: 58ec77ee │ 20│2014-05-09 14:40:40,985 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool │ 20│2014-05-09 14:40:41,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [6b69119f] Correlation ID: 6b69119f, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Etat-Major3. Setting status to Non Responsive. │ 20│2014-05-09 14:40:41,017 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed │al se│2014-05-09 14:40:41,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] Irs placed on server 049943eb-2bcc-4167-a780-7ef76a1f95e9 failed. Proceed Failover │ 20│2014-05-09 14:40:41,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] hostFromVds::selectedVds - serv-vm-adm16, spmStatus Free, storage pool Etat-Major3 │ 20│2014-05-09 14:40:41,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] starting spm on vds serv-vm-adm16, storage pool Etat-Major3, prevId -1, LVER -1 │ 20│2014-05-09 14:40:41,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] START, SpmStartVDSCommand(HostName = serv-vm-adm16, HostId = 13a2bc0a-979a-4fcd-8597-06131030d9a0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFenci│ 20│ng=false), log id: 67d013a4 │ 20│2014-05-09 14:40:41,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] spmStart polling started: taskId = 1046fd3e-71e4-4fcd-bbd0-f17cd6dc08e4 │ 20│2014-05-09 14:40:44,438 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] Failed in HSMGetTaskStatusVDS method -- Nicolas Ecarnot
Same here! An reboot of the nfs machine hosting Export domain, caused the same..Upgrading to 3.4.1 on F19 and then some tweking on my storage domain Gluster based , which mean time became " split-brain" - some brick remove then add and so on - finally solved it. On Fri, May 9, 2014 at 4:55 PM, Nicolas Ecarnot <nicolas@ecarnot.net> wrote:
Hi,
On our second oVirt setup in 3.4.0-1.el6 (that was running fine), I did a yum upgrade on the engine (...sigh...). Then rebooted the engine. This machine is hosting the NFS export domain. Though the VM are still running, the storage domain is in invalid status. You'll find below the engine.log.
At first sight, I thought it was the same issue as : http://lists.ovirt.org/pipermail/users/2014-March/022161.html because it looked very similar. But the NFS export domain connection seemed OK (tested). I did try every trick I could thought of, restarting, checking anything... Our cluster stayed in a broken state.
On second sight, I saw that when rebooting the engine, then NFS export domain was not mounted correctly (I wrote a static /dev/sd-something in fstab, and the iscsi manager changed the letter. Next time, I'll use LVM or a label). So the NFS served was void/empty/black hole.
I just realized all the above, and spent my afternoon in cold sweat. Correcting the NFS mounting and restarting the engine did the trick. What still disturbs me is that the unavailability of the NFS export domain should NOT be a reason for the MASTER storage domain to break!
Following the URL above and the BZ opened by the user ( https://bugzilla.redhat.com/show_bug.cgi?id=1072900), I see this has been corrected in 3.4.1. What gives a perfectly connected NFS export domain, but empty?
PS : I see no 3.4.1 update on CentOS repo.
Regards,
--------------------------
The engine log :
2014-05-09 14:40:37,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] spmStart polling started: taskId = 6d612398-fdad-49f2-9874-5f32a9bf87e2 │ 20│2014-05-09 14:40:40,848 ERROR [org.ovirt.engine.core. vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] Failed in HSMGetTaskStatusVDS method │ 20│2014-05-09 14:40:40,850 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] spmStart polling ended: taskId = 6d612398-fdad-49f2-9874-5f32a9bf87e2 task status = finished │ 20│2014-05-09 14:40:40,850 ERROR [org.ovirt.engine.core. vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 │ 20│2014-05-09 14:40:40,913 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] spmStart polling ended, spm status: Free │ 20│2014-05-09 14:40:40,932 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] START, HSMClearTaskVDSCommand(HostName = serv-vm-adm17, HostId = 049943eb-2bcc-4167-a780-7ef76a1f95e9, taskId=6d612398-fdad-49f2-9874-5f32a9bf87e2), log id: 5cfdc8ce │ 20│2014-05-09 14:40:40,982 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] FINISH, HSMClearTaskVDSCommand, log id: 5cfdc8ce
│ 20│2014-05-09 14:40:40,983 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [f685ea4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common. businessentities.SpmStatusResult@39471ba9, log id: 58ec77ee │ 20│2014-05-09 14:40:40,985 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool │ 20│2014-05-09 14:40:41,009 INFO [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [6b69119f] Correlation ID: 6b69119f, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Etat-Major3. Setting status to Non Responsive. │ 20│2014-05-09 14:40:41,017 ERROR [org.ovirt.engine.core. vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed │al se│2014-05-09 14:40:41,112 INFO [org.ovirt.engine.core. vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] Irs placed on server 049943eb-2bcc-4167-a780-7ef76a1f95e9 failed. Proceed Failover │ 20│2014-05-09 14:40:41,206 INFO [org.ovirt.engine.core. vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] hostFromVds::selectedVds - serv-vm-adm16, spmStatus Free, storage pool Etat-Major3 │ 20│2014-05-09 14:40:41,209 INFO [org.ovirt.engine.core. vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] starting spm on vds serv-vm-adm16, storage pool Etat-Major3, prevId -1, LVER -1 │ 20│2014-05-09 14:40:41,227 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] START, SpmStartVDSCommand(HostName = serv-vm-adm16, HostId = 13a2bc0a-979a-4fcd-8597-06131030d9a0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFenci│ 20│ng=false), log id: 67d013a4
│ 20│2014-05-09 14:40:41,292 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] spmStart polling started: taskId = 1046fd3e-71e4-4fcd-bbd0-f17cd6dc08e4
│ 20│2014-05-09 14:40:44,438 ERROR [org.ovirt.engine.core. vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [6b69119f] Failed in HSMGetTaskStatusVDS method
-- Nicolas Ecarnot _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
2014-05-09 15:55 GMT+02:00 Nicolas Ecarnot <nicolas@ecarnot.net>:
Hi,
On our second oVirt setup in 3.4.0-1.el6 (that was running fine), I did a yum upgrade on the engine (...sigh...). Then rebooted the engine. This machine is hosting the NFS export domain. Though the VM are still running, the storage domain is in invalid status. You'll find below the engine.log.
At first sight, I thought it was the same issue as : http://lists.ovirt.org/pipermail/users/2014-March/022161.html because it looked very similar. But the NFS export domain connection seemed OK (tested). I did try every trick I could thought of, restarting, checking anything... Our cluster stayed in a broken state.
On second sight, I saw that when rebooting the engine, then NFS export domain was not mounted correctly (I wrote a static /dev/sd-something in fstab, and the iscsi manager changed the letter. Next time, I'll use LVM or a label). So the NFS served was void/empty/black hole.
I just realized all the above, and spent my afternoon in cold sweat. Correcting the NFS mounting and restarting the engine did the trick. What still disturbs me is that the unavailability of the NFS export domain should NOT be a reason for the MASTER storage domain to break!
Following the URL above and the BZ opened by the user (https://bugzilla.redhat.com/show_bug.cgi?id=1072900), I see this has been corrected in 3.4.1. What gives a perfectly connected NFS export domain, but empty?
Hi, sorry for jumping late on an old thread, I'm the one reporting that bug. I have two things to say: - taking advantage of a rare opportunity to turn off my production cluster I put it back in that critical situation and I can confirm that with oVirt 3.4.1 the problem has been solved.
PS : I see no 3.4.1 update on CentOS repo.
- me too, until I installed ovirt-release34.rpm (see http://www.ovirt.org/OVirt_3.4.1_release_notes ). All went smooth after that. Best Regards, Giorgio.
Le 14/05/2014 15:36, Giorgio Bersano a écrit :
Following the URL above and the BZ opened by the user (https://bugzilla.redhat.com/show_bug.cgi?id=1072900), I see this has been corrected in 3.4.1. What gives a perfectly connected NFS export domain, but empty?
Hi, sorry for jumping late on an old thread, I'm the one reporting that bug. I have two things to say: - taking advantage of a rare opportunity to turn off my production cluster I put it back in that critical situation and I can confirm that with oVirt 3.4.1 the problem has been solved.
PS : I see no 3.4.1 update on CentOS repo.
- me too, until I installed ovirt-release34.rpm (see http://www.ovirt.org/OVirt_3.4.1_release_notes ). All went smooth after that.
Best Regards, Giorgio.
Thank you Giorgio for taking the time to reply on this. In the end, I understand I have to upgrade, and I will. To the whole team : Looking at the rpm installed on my two oVirt setups, I see that the installed packages are : ovirt-release-el6-8-1.noarch (on CentOS 6.4) and ovirt-release-el6-10.0.1-3.noarch (on CentOS 6.5) The OS are CentOS A "yum search" is showing me that ovirt-release.noarch is available. How do these packages differ? Can I install the ovirt-release.noarch instead of the packages above? Are they aliases? -- Nicolas Ecarnot
This bit me as well (for Fedora, however). There are a whole bunch of ovirt-release variants lying around in various parts of the repository that should be cleaned up as they are merely "attractive nuisances" (http://en.wikipedia.org/wiki/Attractive_nuisance_doctrine). Particularly since the Fedora link (at least) in the Quick Start guide is invalid, so it tempts people to look around and use what they find that looks plausible to somebody who doesn't know any better :) It would be helpful if somebody would: 1. Clean up the hazardous versions of ovirt-release scattered throughout 2. Fix the Quick Start Guide links to point someplace useful for ovirt-release Thanks, Bob On 05/14/2014 10:33 AM, Nicolas Ecarnot wrote:
Le 14/05/2014 15:36, Giorgio Bersano a écrit :
Following the URL above and the BZ opened by the user (https://bugzilla.redhat.com/show_bug.cgi?id=1072900), I see this has been corrected in 3.4.1. What gives a perfectly connected NFS export domain, but empty?
Hi, sorry for jumping late on an old thread, I'm the one reporting that bug. I have two things to say: - taking advantage of a rare opportunity to turn off my production cluster I put it back in that critical situation and I can confirm that with oVirt 3.4.1 the problem has been solved.
PS : I see no 3.4.1 update on CentOS repo.
- me too, until I installed ovirt-release34.rpm (see http://www.ovirt.org/OVirt_3.4.1_release_notes ). All went smooth after that.
Best Regards, Giorgio.
Thank you Giorgio for taking the time to reply on this.
In the end, I understand I have to upgrade, and I will.
To the whole team : Looking at the rpm installed on my two oVirt setups, I see that the installed packages are :
ovirt-release-el6-8-1.noarch (on CentOS 6.4) and ovirt-release-el6-10.0.1-3.noarch (on CentOS 6.5)
The OS are CentOS A "yum search" is showing me that ovirt-release.noarch is available. How do these packages differ? Can I install the ovirt-release.noarch instead of the packages above? Are they aliases?
On 05/14/2014 10:47 AM, Bob Doolittle wrote:
This bit me as well (for Fedora, however).
There are a whole bunch of ovirt-release variants lying around in various parts of the repository that should be cleaned up as they are merely "attractive nuisances" (http://en.wikipedia.org/wiki/Attractive_nuisance_doctrine).
Particularly since the Fedora link (at least) in the Quick Start guide is invalid, so it tempts people to look around and use what they find that looks plausible to somebody who doesn't know any better :)
It would be helpful if somebody would: 1. Clean up the hazardous versions of ovirt-release scattered throughout 2. Fix the Quick Start Guide links to point someplace useful for ovirt-release
can you please open a bug with details on each of these for tracking? thanks, Itamar
Thanks, Bob
On 05/14/2014 10:33 AM, Nicolas Ecarnot wrote:
Le 14/05/2014 15:36, Giorgio Bersano a écrit :
Following the URL above and the BZ opened by the user (https://bugzilla.redhat.com/show_bug.cgi?id=1072900), I see this has been corrected in 3.4.1. What gives a perfectly connected NFS export domain, but empty?
Hi, sorry for jumping late on an old thread, I'm the one reporting that bug. I have two things to say: - taking advantage of a rare opportunity to turn off my production cluster I put it back in that critical situation and I can confirm that with oVirt 3.4.1 the problem has been solved.
PS : I see no 3.4.1 update on CentOS repo.
- me too, until I installed ovirt-release34.rpm (see http://www.ovirt.org/OVirt_3.4.1_release_notes ). All went smooth after that.
Best Regards, Giorgio.
Thank you Giorgio for taking the time to reply on this.
In the end, I understand I have to upgrade, and I will.
To the whole team : Looking at the rpm installed on my two oVirt setups, I see that the installed packages are :
ovirt-release-el6-8-1.noarch (on CentOS 6.4) and ovirt-release-el6-10.0.1-3.noarch (on CentOS 6.5)
The OS are CentOS A "yum search" is showing me that ovirt-release.noarch is available. How do these packages differ? Can I install the ovirt-release.noarch instead of the packages above? Are they aliases?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 05/14/2014 02:28 PM, Itamar Heim wrote:
On 05/14/2014 10:47 AM, Bob Doolittle wrote:
It would be helpful if somebody would: 1. Clean up the hazardous versions of ovirt-release scattered throughout 2. Fix the Quick Start Guide links to point someplace useful for ovirt-release can you please open a bug with details on each of these for tracking?
Done. 1. https://bugzilla.redhat.com/show_bug.cgi?id=1097874 2. https://bugzilla.redhat.com/show_bug.cgi?id=1097872 Note that there is no decent bug category for website/repository so 1097874's assignment is a bit random. I suppose I should file a bug for a new category, but what category would I file it under ;) -Bob
participants (5)
-
Bob Doolittle -
Gabi C -
Giorgio Bersano -
Itamar Heim -
Nicolas Ecarnot