Hello all, last September I deployed Ovirt on a CentOS 8 server, with storage on our gluster (replica 3). I then added some VMs, etc. A few days ago I managed to screw everything up and, after bannging my head for a couple of days, decided to start from scratch. I made a copy of all the data under the storage to a safe space, the ran ovirt-hosted-engine-cleanup and deleted everything under the storage and tried to create a new hosted engine (I tried both from the cockpit and from command line). Everything seems to work fine (I can ssh to the engine) until it tries to save the engine to storage and it fails with the error: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain's metadata]\". HTTP response code is 400."} I don't get any more details. I'm using exactly the same parameters I used before. I have no problems reaching the gluster storage and the process does create the top-level directory and <top-level>/dom_md/ids with the correct ownership. I looked at the glusterfs log files, including the rhev-data-center-mnt-glusterSD-<host:_root>.log file, but I don't spot any specific error. What am I doing wrong ? Is there something else I need to clean up before trying a new deployment ? Should I just try to delete all of the ovirt configuration files ? Which ones ? Thanks, -- As a result of Coronavirus-related precautions, NYU and the Center for Brain Imaging operations will be managed remotely until further notice. All telephone calls and e-mail correspondence are being monitored remotely during our normal business hours of 9am-5pm, Monday through Friday. For MRI scanner-related emergency, please contact: Keith Sanzenbach at keith.sanzenbach@nyu.edu and/or Pablo Velasco at pablo.velasco@nyu.edu For computer/hardware/software emergency, please contact: Valerio Luccio at valerio.luccio@nyu.edu For TMS/EEG-related emergency, please contact: Chrysa Papadaniil at chrysa@nyu.edu For CBI-related administrative emergency, please contact: Jennifer Mangan at jennifer.mangan@nyu.edu Valerio Luccio (212) 998-8736 Center for Brain Imaging 4 Washington Place, Room 158 New York University New York, NY 10003 "In an open world, who needs windows or gates ?"
Forgot to add, it's Ovirt 4.4.5 and the issues seem to have started after I upgraded from 4.4.1. Should I try to roll back ? On 3/26/21 2:44 PM, Valerio Luccio wrote:
Hello all,
last September I deployed Ovirt on a CentOS 8 server, with storage on our gluster (replica 3). I then added some VMs, etc. A few days ago I managed to screw everything up and, after bannging my head for a couple of days, decided to start from scratch.
I made a copy of all the data under the storage to a safe space, the ran ovirt-hosted-engine-cleanup and deleted everything under the storage and tried to create a new hosted engine (I tried both from the cockpit and from command line). Everything seems to work fine (I can ssh to the engine) until it tries to save the engine to storage and it fails with the error:
FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain's metadata]\". HTTP response code is 400."}
I don't get any more details.
I'm using exactly the same parameters I used before. I have no problems reaching the gluster storage and the process does create the top-level directory and <top-level>/dom_md/ids with the correct ownership. I looked at the glusterfs log files, including the rhev-data-center-mnt-glusterSD-<host:_root>.log file, but I don't spot any specific error.
What am I doing wrong ? Is there something else I need to clean up before trying a new deployment ? Should I just try to delete all of the ovirt configuration files ? Which ones ?
Thanks,
-- As a result of Coronavirus-related precautions, NYU and the Center for Brain Imaging operations will be managed remotely until further notice. All telephone calls and e-mail correspondence are being monitored remotely during our normal business hours of 9am-5pm, Monday through Friday. For MRI scanner-related emergency, please contact: Keith Sanzenbach at keith.sanzenbach@nyu.edu and/or Pablo Velasco at pablo.velasco@nyu.edu For computer/hardware/software emergency, please contact: Valerio Luccio at valerio.luccio@nyu.edu For TMS/EEG-related emergency, please contact: Chrysa Papadaniil at chrysa@nyu.edu For CBI-related administrative emergency, please contact: Jennifer Mangan at jennifer.mangan@nyu.edu Valerio Luccio (212) 998-8736 Center for Brain Imaging 4 Washington Place, Room 158 New York University New York, NY 10003 "In an open world, who needs windows or gates ?"
Usually, you should cleanup the engine's volume or even use a new one. Have you tried to write a file as the vdsm user ? Best Regards,Strahil Nikolov On Fri, Mar 26, 2021 at 21:49, Valerio Luccio<valerio.luccio@nyu.edu> wrote: _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/I6LWBFDPLCAJWQ...
Thanks Strahil, I did do a cleanup first. The wizard creates the top level directory and the 'dom_md' subdirectory and the ids file as the vdsm user on the gluster storage.
So it does turn out that I have some issue with gluster. I was fooled by seeing that the wizard does create the top-level directory and the ids file, but if I try to create a file from command line I get an error. I don't have an issue from other servers, so it's not a problem with the gluster storage. Nice catch Strahil, you pointed me in the right direction. In the latest update, which I did Friday (not part of my initial ovirt update), it upgraded gluster from 7 to 8. What happens if I roll these back. Is gluster 8 really necessary at this point ? I have some legacy systems that use that gluster storage and I'm afraid of breaking them.
On Sunday, 28 March 2021 06:49:18 CEST valerio.luccio@nyu.edu wrote:
So it does turn out that I have some issue with gluster. I was fooled by seeing that the wizard does create the top-level directory and the ids file, but if I try to create a file from command line I get an error. I don't have an issue from other servers, so it's not a problem with the gluster storage.
Nice catch Strahil, you pointed me in the right direction.
In the latest update, which I did Friday (not part of my initial ovirt update), it upgraded gluster from 7 to 8. What happens if I roll these back. Is gluster 8 really necessary at this point ?
I don't think this is required by oVirt, e.g. in vdsm spec file I can see we require version 6.0 [1]. Maybe was upgraded during general OS upgrade? [1] https://github.com/oVirt/vdsm/blob/master/vdsm.spec.in#L25
I have some legacy systems that use that gluster storage and I'm afraid of breaking them. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2QA5DN47ZIHSC PCBTQTQ3Z6WG5NKSJF3/
From my dnf history: Upgrade glusterfs-8.4-1.el8.x86_64 @ovirt-4.4-centos-gluster8 But if all that is required is v. 6 I should be able to safely downgrade to 7.9. Thanks for the spec reference.
On Sunday, 28 March 2021 06:49:18 CEST valerio.luccio(a)nyu.edu wrote:
I don't think this is required by oVirt, e.g. in vdsm spec file I can see we require version 6.0 [1]. Maybe was upgraded during general OS upgrade?
[1] https://github.com/oVirt/vdsm/blob/master/vdsm.spec.in#L25
OK, I downgraded glusterfs to 7.9 (got the RPMs from the CentOS mirror and did a "dnf --allowerasing downgrade <RPMs>") and I was able to deploy a new engine. Now on to solve all my other problems ;-(
On Monday, 29 March 2021 16:20:38 CEST valerio.luccio@nyu.edu wrote:
OK, I downgraded glusterfs to 7.9 (got the RPMs from the CentOS mirror and did a "dnf --allowerasing downgrade <RPMs>") and I was able to deploy a new engine.
Now on to solve all my other problems ;-(
do you still fail to create SD metadata? If so, why it fails when you try manually from cmd line?
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/7TBEXM4XYNFCF ITYXL3CCLPDSQ77HHQK/
Can you share the gluster brick logs (when the peoblems were happening) ? I think that it's important to solve this issue. If you wish, we can move the topic to gluster-users mailing list. Best Regards,Strahil Nikolov It works, it doesn't fail. Thanks.
do you still fail to create SD metadata? If so, why it fails when you try manually from cmd line?
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZUVDGVLEZMEBXA...
Strahil, I did share the logs on the gluster-users mailing list and you replied and included gluster-devel in recipients. Are you looking for other logs ? On 3/31/21 4:29 PM, Strahil Nikolov wrote:
Can you share the gluster brick logs (when the peoblems were happening) ?
I think that it's important to solve this issue.
If you wish, we can move the topic to gluster-users mailing list.
Best Regards, Strahil Nikolov
It works, it doesn't fail.
Thanks.
> > do you still fail to create SD metadata? If so, why it fails when you try > manually from cmd line? _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZUVDGVLEZMEBXA... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZUVDGVLEZMEBXAPH563ZMPYDGNQM5JSL/>
-- As a result of Coronavirus-related precautions, NYU and the Center for Brain Imaging operations will be managed remotely until further notice. All telephone calls and e-mail correspondence are being monitored remotely during our normal business hours of 9am-5pm, Monday through Friday. For MRI scanner-related emergency, please contact: Keith Sanzenbach at keith.sanzenbach@nyu.edu and/or Pablo Velasco at pablo.velasco@nyu.edu For computer/hardware/software emergency, please contact: Valerio Luccio at valerio.luccio@nyu.edu For TMS/EEG-related emergency, please contact: Chrysa Papadaniil at chrysa@nyu.edu For CBI-related administrative emergency, please contact: Jennifer Mangan at jennifer.mangan@nyu.edu Valerio Luccio (212) 998-8736 Center for Brain Imaging 4 Washington Place, Room 158 New York University New York, NY 10003 "In an open world, who needs windows or gates ?"
participants (4)
-
Strahil Nikolov -
Valerio Luccio -
valerio.luccio＠nyu.edu
-
Vojtech Juranek