Big problem on Ovirt 4.4
Hi, I run a small test node and engine of ovirt since 4.3. Its in my home, but I kind messed up and now I can't use or reinstall it without loosing all my vm's data (at least this is the point my knowledge of ovirt is). I got the issue where I could not update yum since CentOS 8 mirrors would not answer. And I tried some solution I found in the list archives, and got really bad here. I had a power outage last Saturday and my node got to nonresponsive. No matter what I did, never left this state. Kept complaining about the CPU Type, I run it on Ryzen 3400G (used to run on Intel i5, also desktop CPU, and never had an issue, when I changed to Ryzen, every outage from power here is a problem), and the node wouldn't come back. So I tried to upgrade my cluster to 4.4 level (was 4.3) and it didn't help. So I tried to get back to 4.3 and a message said just empty clusters could do that. I then removed my ryzen node from it and it got back to 4.3.But now I could not install the node. I tried changing its name, no good. Failed on how to get packages from GLuster8 repo.That is where my story got sad. I tried to update my repo using the steps in the last post from https://lists.ovirt.org/archives/list/users@ovirt.org/thread/D3FEXQ5KOLN3SST.... Bad idea, and another one was I didn't do a backup of the engine. Now my engine won't start, the backup won't run: engine-backup --mode=backup --file=/root/backup_deuruim --log=/root/backup_deuruim.logStart of engine-backup with mode 'backup'scope: allarchive file: /root/backup_deuruimlog file: /root/backup_deuruim.logBacking up:Notifying engineNotifying engineFATAL: Failed notifying engine pg_dump: error: Dumping the contents of table "audit_log" failed: PQgetResult() failed.pg_dump: error: Error message from server: ERROR: MultiXactId 1076887572 has not been created yet -- apparent wraparoundpg_dump: error: The command was: COPY public.audit_log (audit_log_id, user_id, user_name, vm_id, vm_name, vm_template_id, vm_template_name, vds_id, vds_name, log_time, log_type_name, log_type, severity, message, processed, storage_pool_id, storage_pool_name, storage_domain_id, storage_domain_name, cluster_id, cluster_name, correlation_id, job_id, quota_id, quota_name, gluster_volume_id, gluster_volume_name, origin, custom_event_id, event_flood_in_sec, custom_data, deleted, call_stack, brick_id, brick_path, custom_id) TO stdout;2022-03-24 13:16:10 3461: FATAL: Database engine backup failed the Admin portal won't work, and I get a message on the login screen from engine saying I have a web console at port 9090. I don't get it. As my last resort I am here, trying at least to know a way I could backup it and install the engine elsewhere. All vm data is fine, but as the names are cryptographic, import them would have me guessing things, right? The number of VM's is low, 6 or 7, so if there is a way to do it manually, I would try. Installing it again would be my nightmare here. thanks, matheus ps: if needed, I can provide more info.
On 24. 3. 2022, at 18:38, Nenhum de Nos via Users <users@ovirt.org> wrote:
Hi,
I run a small test node and engine of ovirt since 4.3. Its in my home, but I kind messed up and now I can't use or reinstall it without loosing all my vm's data (at least this is the point my knowledge of ovirt is).
I got the issue where I could not update yum since CentOS 8 mirrors would not answer. And I tried some solution I found in the list archives, and got really bad here.
Hi, well, CentOS 8 doesn't exist anymore, you would have to switch to 8-stream
I had a power outage last Saturday and my node got to nonresponsive. No matter what I did, never left this state. Kept complaining about the CPU Type, I run it on Ryzen 3400G (used to run on Intel i5, also desktop CPU, and never had an issue, when I changed to Ryzen, every outage from power here is a problem), and the node wouldn't come back. So I tried to upgrade my cluster to 4.4 level (was 4.3) and it didn't help. So I tried to get back to 4.3 and a message said just empty clusters could do that. I then removed my ryzen node from it and it got back to 4.3. But now I could not install the node. I tried changing its name, no good. Failed on how to get packages from GLuster8 repo. That is where my story got sad. I tried to update my repo using the steps in the last post from https://lists.ovirt.org/archives/list/users@ovirt.org/thread/D3FEXQ5KOLN3SST... <https://lists.ovirt.org/archives/list/users@ovirt.org/thread/D3FEXQ5KOLN3SST3MPGIHSCEF52IBTKY/>. Bad idea, and another one was I didn't do a backup of the engine.
Now my engine won't start, the backup won't run:
engine-backup --mode=backup --file=/root/backup_deuruim --log=/root/backup_deuruim.log Start of engine-backup with mode 'backup' scope: all archive file: /root/backup_deuruim log file: /root/backup_deuruim.log Backing up: Notifying engine Notifying engine FATAL: Failed notifying engine
pg_dump: error: Dumping the contents of table "audit_log" failed: PQgetResult() failed. pg_dump: error: Error message from server: ERROR: MultiXactId 1076887572 has not been created yet -- apparent wraparound pg_dump: error: The command was: COPY public.audit_log (audit_log_id, user_id, user_name, vm_id, vm_name, vm_template_id, vm_template_name, vds_id, vds_name, log_time, log_type_name, log_type, severity, message, processed, storage_pool_id, storage_pool_name, storage_domain_id, storage_domain_name, cluster_id, cluster_name, correlation_id, job_id, quota_id, quota_name, gluster_volume_id, gluster_volume_name, origin, custom_event_id, event_flood_in_sec, custom_data, deleted, call_stack, brick_id, brick_path, custom_id) TO stdout; 2022-03-24 13:16:10 3461: FATAL: Database engine backup failed
the Admin portal won't work, and I get a message on the login screen from engine saying I have a web console at port 9090. I don't get it.
I don't get it either, it all sounds too confusing.
As my last resort I am here, trying at least to know a way I could backup it and install the engine elsewhere. All vm data is fine, but as the names are cryptographic, import them would have me guessing things, right?
The number of VM's is low, 6 or 7, so if there is a way to do it manually, I would try. Installing it again would be my nightmare here.
with just a few VMs it's indeed best to just reinstall from scratch and importing the storage domain back. You would only be missing the other non-VM configuration in oVirt that you have to do again, but the VMs should have the right name and all after import. Assuming you still have the storage domain somewhere safe. Thanks, michal
thanks,
matheus
ps: if needed, I can provide more info.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PVFGIOV6KKG3BP...
On Fri, 25 Mar 2022 12:37:21 +0100 Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 24. 3. 2022, at 18:38, Nenhum de Nos via Users <users@ovirt.org> wrote:
Hi,
I run a small test node and engine of ovirt since 4.3. Its in my home, but I kind messed up and now I can't use or reinstall it without loosing all my vm's data (at least this is the point my knowledge of ovirt is).
I got the issue where I could not update yum since CentOS 8 mirrors would not answer. And I tried some solution I found in the list archives, and got really bad here.
Hi, well, CentOS 8 doesn't exist anymore, you would have to switch to 8-stream
I had a power outage last Saturday and my node got to nonresponsive. No matter what I did, never left this state. Kept complaining about the CPU Type, I run it on Ryzen 3400G (used to run on Intel i5, also desktop CPU, and never had an issue, when I changed to Ryzen, every outage from power here is a problem), and the node wouldn't come back. So I tried to upgrade my cluster to 4.4 level (was 4.3) and it didn't help. So I tried to get back to 4.3 and a message said just empty clusters could do that. I then removed my ryzen node from it and it got back to 4.3. But now I could not install the node. I tried changing its name, no good. Failed on how to get packages from GLuster8 repo. That is where my story got sad. I tried to update my repo using the steps in the last post from https://lists.ovirt.org/archives/list/users@ovirt.org/thread/D3FEXQ5KOLN3SST... <https://lists.ovirt.org/archives/list/users@ovirt.org/thread/D3FEXQ5KOLN3SST3MPGIHSCEF52IBTKY/>. Bad idea, and another one was I didn't do a backup of the engine.
Now my engine won't start, the backup won't run:
engine-backup --mode=backup --file=/root/backup_deuruim --log=/root/backup_deuruim.log Start of engine-backup with mode 'backup' scope: all archive file: /root/backup_deuruim log file: /root/backup_deuruim.log Backing up: Notifying engine Notifying engine FATAL: Failed notifying engine
pg_dump: error: Dumping the contents of table "audit_log" failed: PQgetResult() failed. pg_dump: error: Error message from server: ERROR: MultiXactId 1076887572 has not been created yet -- apparent wraparound pg_dump: error: The command was: COPY public.audit_log (audit_log_id, user_id, user_name, vm_id, vm_name, vm_template_id, vm_template_name, vds_id, vds_name, log_time, log_type_name, log_type, severity, message, processed, storage_pool_id, storage_pool_name, storage_domain_id, storage_domain_name, cluster_id, cluster_name, correlation_id, job_id, quota_id, quota_name, gluster_volume_id, gluster_volume_name, origin, custom_event_id, event_flood_in_sec, custom_data, deleted, call_stack, brick_id, brick_path, custom_id) TO stdout; 2022-03-24 13:16:10 3461: FATAL: Database engine backup failed
the Admin portal won't work, and I get a message on the login screen from engine saying I have a web console at port 9090. I don't get it.
I don't get it either, it all sounds too confusing.
As my last resort I am here, trying at least to know a way I could backup it and install the engine elsewhere. All vm data is fine, but as the names are cryptographic, import them would have me guessing things, right?
The number of VM's is low, 6 or 7, so if there is a way to do it manually, I would try. Installing it again would be my nightmare here.
with just a few VMs it's indeed best to just reinstall from scratch and importing the storage domain back. You would only be missing the other non-VM configuration in oVirt that you have to do again, but the VMs should have the right name and all after import. Assuming you still have the storage domain somewhere safe.
Thanks, michal
Hi Michal, first thanks for the answer. I do have the storage domain intact, and I kinda already did that of reinstalling it all from scratch. But I hit another wall here, I can't add my ryzen node to a cluster, every option I choose for CPU Type it get refused, so I never got to test the storage domain import yet :( On cluster 4.3 it worked a little unstable using one option that now is missing when the cluster is 4.6 level. Is my Ryzen 3400G usable on ovirt? So far I can't find a way for it to be :( thanks again, matheus
Even if you start with brand new engine you can still import the storage domains and the VMs inside it.Drawback -> you might have to delete any VM snapshots prior the import. Best Regards,Strahil Nikolov On Fri, Mar 25, 2022 at 13:40, Michal Skrivanek<michal.skrivanek@redhat.com> wrote: _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KW6NQOM2JOBFXK...
On Fri, 25 Mar 2022 18:01:01 +0000 (UTC) Strahil Nikolov via Users <users@ovirt.org> wrote:
Even if you start with brand new engine you can still import the storage domains and the VMs inside it.Drawback -> you might have to delete any VM snapshots prior the import. Best Regards,Strahil Nikolov
Hi Strahil, thanks for the info. My main issue now is the ovirt node. Its no easy for me to get an intel based one and my ryzen based I cannot add to any cluster. I tried creating clusters from 4.2 to 4.6, and its the same error over and over again. Not compatible :( I don't get it, before last satyrday it was, and worked fine. I reinstalled all, I need those VM's ASAP, most my things are offline for a week now. It's house stuff, yes, not big company production VM's, no, but I need them to have all my stuff here. thanks to all that helped. matheus
Can you summarize the situation, as it's hard to get an idea of the situation? On the Hypervisor: rpm -qa | grep ovirt | sortvirsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf domcapabilities From the Engine UI:Cluster version and CPU type From the HostedEngine VM:rpm -qa | grep ovirt | sort Best Regards,Strahil Nikolov On Fri, Mar 25, 2022 at 20:31, Nenhum_de_Nos<lojas@arroway.org> wrote: On Fri, 25 Mar 2022 18:01:01 +0000 (UTC) Strahil Nikolov via Users <users@ovirt.org> wrote:
Even if you start with brand new engine you can still import the storage domains and the VMs inside it.Drawback -> you might have to delete any VM snapshots prior the import. Best Regards,Strahil Nikolov
Hi Strahil, thanks for the info. My main issue now is the ovirt node. Its no easy for me to get an intel based one and my ryzen based I cannot add to any cluster. I tried creating clusters from 4.2 to 4.6, and its the same error over and over again. Not compatible :( I don't get it, before last satyrday it was, and worked fine. I reinstalled all, I need those VM's ASAP, most my things are offline for a week now. It's house stuff, yes, not big company production VM's, no, but I need them to have all my stuff here. thanks to all that helped. matheus
On Sat, 26 Mar 2022 06:37:39 +0000 (UTC) Strahil Nikolov via Users <users@ovirt.org> wrote:
Can you summarize the situation, as it's hard to get an idea of the situation? On the Hypervisor: rpm -qa | grep ovirt | sortvirsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf domcapabilities From the Engine UI:Cluster version and CPU type From the HostedEngine VM:rpm -qa | grep ovirt | sort Best Regards, Nikolov
Hi Strahil, I know it was not the best explanation of the issue, I was kind stressed as my VM's were offline for almost a week. Now things are quite again. I think this problem I went through is not quite common to ovirt users as by what I could see here, people use server hardware, unlike me. After the power outage I could not get my node back active on the cluster (there is just one node here). And no matter how many times I tried, the logs kept saying the CPU was not compatible. I am answering here as I could not find this on my search through google and mail lists, forums (just could not see the redhat forum, I don't have account there). My problem was that my machine got some kind of bios reset, at least it disabled Virtualization Capabilities. And as I had not turned it off, I never would suspect it was this. In a desperate move, I installed ESXi on the machine and begun migrating my VM's. When the first was able to run, he hypervisor told me mu CPU was not VM capable. So then it hit me. I got back on the ovirt engine I had installed from scratch, and it was fine installing the node on cluster version 4.3 and the AMD EPYC IBPB SSBD. Then I just had to follow your hint to import the Storage Domain (another issue here, I tried to add it and then canceled, so it got stuck and couldn't import it until I wen to psql and delete the line by hand). The error message was always "Host ryzen-node moved to Non-Operational state as host CPU type is not supported in this cluster compatibility version or is not supported at all". And I never would guess that above from this :( When I tried to remove the node and add again, as it was CentOS8 based and I could not install packages, it all went downhill to a state where I tried to update packages from command line and the Engine GUI would not work anymore. Now the storm is behind me and I now have both node and engine up to date. I will now seek information on how to keep them updated so I don't have to be on that road again. Thanks for all support! I guess those rpm commands are no longer needed, right? If so, I can send afterwards. One last question, is it safe to update the cluster version to 4.6 using Ryzen 3400G? thanks, matheus
On Fri, Mar 25, 2022 at 20:31, Nenhum_de_Nos<lojas@arroway.org> wrote: On Fri, 25 Mar 2022 18:01:01 +0000 (UTC) Strahil Nikolov via Users <users@ovirt.org> wrote:
Even if you start with brand new engine you can still import the storage domains and the VMs inside it.Drawback -> you might have to delete any VM snapshots prior the import. Best Regards,Strahil Nikolov
Hi Strahil, thanks for the info.
My main issue now is the ovirt node. Its no easy for me to get an intel based one and my ryzen based I cannot add to any cluster. I tried creating clusters from 4.2 to 4.6, and its the same error over and over again. Not compatible :(
I don't get it, before last satyrday it was, and worked fine. I reinstalled all, I need those VM's ASAP, most my things are offline for a week now. It's house stuff, yes, not big company production VM's, no, but I need them to have all my stuff here.
thanks to all that helped.
matheus
You can reinstall your node - it will be assigned ryzen-based model for the cluster.Once you import the storage domains, all VMs should be visible. Then you can edit each VM, verify NIC, MAC address and change the CPU type of the VM to the one that is available in the Cluster (respectfully the host). Best Regards,Strahil Nikolov On Sat, Mar 26, 2022 at 19:02, Nenhum_de_Nos<lojas@arroway.org> wrote: On Sat, 26 Mar 2022 06:37:39 +0000 (UTC) Strahil Nikolov via Users <users@ovirt.org> wrote:
Can you summarize the situation, as it's hard to get an idea of the situation? On the Hypervisor: rpm -qa | grep ovirt | sortvirsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf domcapabilities From the Engine UI:Cluster version and CPU type From the HostedEngine VM:rpm -qa | grep ovirt | sort Best Regards, Nikolov
Hi Strahil, I know it was not the best explanation of the issue, I was kind stressed as my VM's were offline for almost a week. Now things are quite again. I think this problem I went through is not quite common to ovirt users as by what I could see here, people use server hardware, unlike me. After the power outage I could not get my node back active on the cluster (there is just one node here). And no matter how many times I tried, the logs kept saying the CPU was not compatible. I am answering here as I could not find this on my search through google and mail lists, forums (just could not see the redhat forum, I don't have account there). My problem was that my machine got some kind of bios reset, at least it disabled Virtualization Capabilities. And as I had not turned it off, I never would suspect it was this. In a desperate move, I installed ESXi on the machine and begun migrating my VM's. When the first was able to run, he hypervisor told me mu CPU was not VM capable. So then it hit me. I got back on the ovirt engine I had installed from scratch, and it was fine installing the node on cluster version 4.3 and the AMD EPYC IBPB SSBD. Then I just had to follow your hint to import the Storage Domain (another issue here, I tried to add it and then canceled, so it got stuck and couldn't import it until I wen to psql and delete the line by hand). The error message was always "Host ryzen-node moved to Non-Operational state as host CPU type is not supported in this cluster compatibility version or is not supported at all". And I never would guess that above from this :( When I tried to remove the node and add again, as it was CentOS8 based and I could not install packages, it all went downhill to a state where I tried to update packages from command line and the Engine GUI would not work anymore. Now the storm is behind me and I now have both node and engine up to date. I will now seek information on how to keep them updated so I don't have to be on that road again. Thanks for all support! I guess those rpm commands are no longer needed, right? If so, I can send afterwards. One last question, is it safe to update the cluster version to 4.6 using Ryzen 3400G? thanks, matheus
On Fri, Mar 25, 2022 at 20:31, Nenhum_de_Nos<lojas@arroway.org> wrote: On Fri, 25 Mar 2022 18:01:01 +0000 (UTC) Strahil Nikolov via Users <users@ovirt.org> wrote:
Even if you start with brand new engine you can still import the storage domains and the VMs inside it.Drawback -> you might have to delete any VM snapshots prior the import. Best Regards,Strahil Nikolov
Hi Strahil, thanks for the info.
My main issue now is the ovirt node. Its no easy for me to get an intel based one and my ryzen based I cannot add to any cluster. I tried creating clusters from 4.2 to 4.6, and its the same error over and over again. Not compatible :(
I don't get it, before last satyrday it was, and worked fine. I reinstalled all, I need those VM's ASAP, most my things are offline for a week now. It's house stuff, yes, not big company production VM's, no, but I need them to have all my stuff here.
thanks to all that helped.
matheus
participants (4)
-
Michal Skrivanek -
Nenhum de Nos -
Nenhum_de_Nos -
Strahil Nikolov