Cinderlib problem after upgrade from 4.3.10 to 4.4.5
Hello all, first of all thank you very much for this stable virtualization environment. It has been a pillar for our company’s business for more than 5 years now and after migrating from version 3 to 4 it has been so stable ever since. Anyway, I ran into a problem I cannot fix on my own yesterday: After a lot of consideration and hesitation since this is a production environment I followed the upgrade guide (https://www.ovirt.org/documentation/upgrade_guide/ <https://www.ovirt.org/documentation/upgrade_guide/>), configured a vanilla CentOS 8 server as controller, decommissioned the old 4.3 controller and fired up the new one. It worked like a charm until I tried to migrate VMs, start new ones or even create new disks. We use Ceph as managed storage, providing a SSD only and a HDD only pool. The UI simply told me that there was an error. I started investigating the issue and found corresponding log entries in ovirt-engine.log: 2021-03-22 10:36:37,247+01 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24) [67bf193c] cinderlib execution failed: But that was all the engine had to say about the issue. There was no stack trace or additional information. There is no logfile in /var/log/ovirt-engine/cinderlib/, the directory simply is empty while on the other controller it was frequently filed with annoying „already mounted“ messages. Can anyone help me with that issue? I searched the web for a solution or someone else with the same problem, but came up empty. Is there a way to turn up the log level for cinderlib? Are there any dependencies I have to install besides the ovirt packages? Any help is very much appreciated! Kind regards and stay healthy, Marc -- ________________________________________________________________________ Dipl.-Inform. Marc-Christian Schröer schroeer@ingenit.com Geschäftsführer / CEO ---------------------------------------------------------------------- ingenit GmbH & Co. KG Tel. +49 (0)231 58 698-120 Emil-Figge-Strasse 76-80 Fax. +49 (0)231 58 698-121 D-44227 Dortmund www.ingenit.com Registergericht: Amtsgericht Dortmund, HRA 13 914 Gesellschafter : Thomas Klute, Marc-Christian Schröer ________________________________________________________________________
If the log is empty it usually means cinderlib-client.py failed during startup, probably because the dependencies are missing. python3-cinderlib is required on the engine machine (and ceph-common, since you use ceph), python3-os-brick is required on the hosts (and ceph-common). See the instructions here for ussuri: https://ovirt.org/documentation/installing_ovirt_as_a_standalone_manager_wit... On Tue, Mar 23, 2021 at 9:20 AM Marc-Christian Schröer <schroeer@ingenit.com> wrote:
Hello all,
first of all thank you very much for this stable virtualization environment. It has been a pillar for our company’s business for more than 5 years now and after migrating from version 3 to 4 it has been so stable ever since. Anyway, I ran into a problem I cannot fix on my own yesterday:
After a lot of consideration and hesitation since this is a production environment I followed the upgrade guide (https://www.ovirt.org/documentation/upgrade_guide/), configured a vanilla CentOS 8 server as controller, decommissioned the old 4.3 controller and fired up the new one. It worked like a charm until I tried to migrate VMs, start new ones or even create new disks. We use Ceph as managed storage, providing a SSD only and a HDD only pool. The UI simply told me that there was an error.
I started investigating the issue and found corresponding log entries in ovirt-engine.log:
2021-03-22 10:36:37,247+01 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24) [67bf193c] cinderlib execution failed:
But that was all the engine had to say about the issue. There was no stack trace or additional information. There is no logfile in /var/log/ovirt-engine/cinderlib/, the directory simply is empty while on the other controller it was frequently filed with annoying „already mounted“ messages.
Can anyone help me with that issue? I searched the web for a solution or someone else with the same problem, but came up empty. Is there a way to turn up the log level for cinderlib? Are there any dependencies I have to install besides the ovirt packages? Any help is very much appreciated!
Kind regards and stay healthy, Marc
-- ________________________________________________________________________
Dipl.-Inform. Marc-Christian Schröer schroeer@ingenit.com Geschäftsführer / CEO ---------------------------------------------------------------------- ingenit GmbH & Co. KG Tel. +49 (0)231 58 698-120 Emil-Figge-Strasse 76-80 Fax. +49 (0)231 58 698-121 D-44227 Dortmund www.ingenit.com
Registergericht: Amtsgericht Dortmund, HRA 13 914 Gesellschafter : Thomas Klute, Marc-Christian Schröer ________________________________________________________________________
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/O22IFE3PSFJ6VM...
Hello all, hello Benny, thank you very much for the helpful answer and pointing me to the documentation. After setting up the server from scratch, installing some dependencies and fixing the issue of a missing __DEFAULT__ volume type in the ovirt_cinderlib database I hit a new bump: psycopg2.ProgrammingError: column volumes.service_uuid does not exist The /var/log/ovirt-engine/cinderlib/cinderlib.log file contains a somewhat lengthy entry of a failed SQL query basically telling me that the database structure is not what is expected by ovirt. I upgraded from 4.3 to 4.4 using the backup/restore upgrade method and would have thought oVirt’s engine-setup had migrated the database to a new model. Do I have to execute some upgrade command manually? Kind regards and thanks for all the help, Marc -- ________________________________________________________________________ Dipl.-Inform. Marc-Christian Schröer schroeer@ingenit.com <mailto:schroeer@ingenit.com> Geschäftsführer / CEO ---------------------------------------------------------------------- ingenit GmbH & Co. KG Tel. +49 (0)231 58 698-120 Emil-Figge-Strasse 76-80 Fax. +49 (0)231 58 698-121 D-44227 Dortmund www.ingenit.com <http://www.ingenit.com/> Registergericht: Amtsgericht Dortmund, HRA 13 914 Gesellschafter : Thomas Klute, Marc-Christian Schröer ________________________________________________________________________
Am 23.03.2021 um 08:38 schrieb Benny Zlotnik <bzlotnik@redhat.com>:
If the log is empty it usually means cinderlib-client.py failed during startup, probably because the dependencies are missing. python3-cinderlib is required on the engine machine (and ceph-common, since you use ceph), python3-os-brick is required on the hosts (and ceph-common). See the instructions here for ussuri: https://ovirt.org/documentation/installing_ovirt_as_a_standalone_manager_wit...
On Tue, Mar 23, 2021 at 9:20 AM Marc-Christian Schröer <schroeer@ingenit.com> wrote:
Hello all,
first of all thank you very much for this stable virtualization environment. It has been a pillar for our company’s business for more than 5 years now and after migrating from version 3 to 4 it has been so stable ever since. Anyway, I ran into a problem I cannot fix on my own yesterday:
After a lot of consideration and hesitation since this is a production environment I followed the upgrade guide (https://www.ovirt.org/documentation/upgrade_guide/), configured a vanilla CentOS 8 server as controller, decommissioned the old 4.3 controller and fired up the new one. It worked like a charm until I tried to migrate VMs, start new ones or even create new disks. We use Ceph as managed storage, providing a SSD only and a HDD only pool. The UI simply told me that there was an error.
I started investigating the issue and found corresponding log entries in ovirt-engine.log:
2021-03-22 10:36:37,247+01 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24) [67bf193c] cinderlib execution failed:
But that was all the engine had to say about the issue. There was no stack trace or additional information. There is no logfile in /var/log/ovirt-engine/cinderlib/, the directory simply is empty while on the other controller it was frequently filed with annoying „already mounted“ messages.
Can anyone help me with that issue? I searched the web for a solution or someone else with the same problem, but came up empty. Is there a way to turn up the log level for cinderlib? Are there any dependencies I have to install besides the ovirt packages? Any help is very much appreciated!
Kind regards and stay healthy, Marc
-- ________________________________________________________________________
Dipl.-Inform. Marc-Christian Schröer schroeer@ingenit.com Geschäftsführer / CEO ---------------------------------------------------------------------- ingenit GmbH & Co. KG Tel. +49 (0)231 58 698-120 Emil-Figge-Strasse 76-80 Fax. +49 (0)231 58 698-121 D-44227 Dortmund www.ingenit.com
Registergericht: Amtsgericht Dortmund, HRA 13 914 Gesellschafter : Thomas Klute, Marc-Christian Schröer ________________________________________________________________________
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/O22IFE3PSFJ6VM...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3EXKE44PBNBTMZ...
There is no upgrade path for cinderlib between 4.3 and 4.4, also in 4.3 cinderlib was installed via pip with the 0.x version, while 4.4 is 1.x. In ovirt-engine we only create the cinderlib database, we don't control the tables, this is controlled by the ORM cinderlib uses. We did get a report about the __DEFAULT__ issue, but I didn't get the bug report I asked for, so please submit a bug with all the details so we can properly investigate and provide some solution. On Thu, Mar 25, 2021 at 9:13 AM Marc-Christian Schröer <schroeer@ingenit.com> wrote:
Hello all, hello Benny,
thank you very much for the helpful answer and pointing me to the documentation. After setting up the server from scratch, installing some dependencies and fixing the issue of a missing __DEFAULT__ volume type in the ovirt_cinderlib database I hit a new bump:
psycopg2.ProgrammingError: column volumes.service_uuid does not exist
The /var/log/ovirt-engine/cinderlib/cinderlib.log file contains a somewhat lengthy entry of a failed SQL query basically telling me that the database structure is not what is expected by ovirt. I upgraded from 4.3 to 4.4 using the backup/restore upgrade method and would have thought oVirt’s engine-setup had migrated the database to a new model.
Do I have to execute some upgrade command manually?
Kind regards and thanks for all the help, Marc
-- ________________________________________________________________________
Dipl.-Inform. Marc-Christian Schröer schroeer@ingenit.com Geschäftsführer / CEO ---------------------------------------------------------------------- ingenit GmbH & Co. KG Tel. +49 (0)231 58 698-120 Emil-Figge-Strasse 76-80 Fax. +49 (0)231 58 698-121 D-44227 Dortmund www.ingenit.com
Registergericht: Amtsgericht Dortmund, HRA 13 914 Gesellschafter : Thomas Klute, Marc-Christian Schröer ________________________________________________________________________
Am 23.03.2021 um 08:38 schrieb Benny Zlotnik <bzlotnik@redhat.com>:
If the log is empty it usually means cinderlib-client.py failed during startup, probably because the dependencies are missing. python3-cinderlib is required on the engine machine (and ceph-common, since you use ceph), python3-os-brick is required on the hosts (and ceph-common). See the instructions here for ussuri: https://ovirt.org/documentation/installing_ovirt_as_a_standalone_manager_wit...
On Tue, Mar 23, 2021 at 9:20 AM Marc-Christian Schröer <schroeer@ingenit.com> wrote:
Hello all,
first of all thank you very much for this stable virtualization environment. It has been a pillar for our company’s business for more than 5 years now and after migrating from version 3 to 4 it has been so stable ever since. Anyway, I ran into a problem I cannot fix on my own yesterday:
After a lot of consideration and hesitation since this is a production environment I followed the upgrade guide (https://www.ovirt.org/documentation/upgrade_guide/), configured a vanilla CentOS 8 server as controller, decommissioned the old 4.3 controller and fired up the new one. It worked like a charm until I tried to migrate VMs, start new ones or even create new disks. We use Ceph as managed storage, providing a SSD only and a HDD only pool. The UI simply told me that there was an error.
I started investigating the issue and found corresponding log entries in ovirt-engine.log:
2021-03-22 10:36:37,247+01 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24) [67bf193c] cinderlib execution failed:
But that was all the engine had to say about the issue. There was no stack trace or additional information. There is no logfile in /var/log/ovirt-engine/cinderlib/, the directory simply is empty while on the other controller it was frequently filed with annoying „already mounted“ messages.
Can anyone help me with that issue? I searched the web for a solution or someone else with the same problem, but came up empty. Is there a way to turn up the log level for cinderlib? Are there any dependencies I have to install besides the ovirt packages? Any help is very much appreciated!
Kind regards and stay healthy, Marc
-- ________________________________________________________________________
Dipl.-Inform. Marc-Christian Schröer schroeer@ingenit.com Geschäftsführer / CEO ---------------------------------------------------------------------- ingenit GmbH & Co. KG Tel. +49 (0)231 58 698-120 Emil-Figge-Strasse 76-80 Fax. +49 (0)231 58 698-121 D-44227 Dortmund www.ingenit.com
Registergericht: Amtsgericht Dortmund, HRA 13 914 Gesellschafter : Thomas Klute, Marc-Christian Schröer ________________________________________________________________________
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/O22IFE3PSFJ6VM...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3EXKE44PBNBTMZ...
participants (2)
-
Benny Zlotnik -
Marc-Christian Schröer