Cinderlib died after upgrade Cluster Compatibility version 4.4 ->4.5
hi all, after upgrade oVirt 4.4.7 Cluster Compatibility version 4.4 ->4.5 cinderlib storage is died: tail -f /var/log/ovirt-engine/cinderlib/cinderlib.log 2021-01-20 12:35:28,781 - cinderlib-client - ERROR - Failure occurred when trying to run command 'delete_volume': Volume type with name __DEFAULT__ could not be found. [3d4a9aa9-c5f0-498c-b650-bc64d979f194] 2021-01-20 12:43:31,209 - cinderlib-client - ERROR - Failure occurred when trying to run command 'delete_volume': Volume type with name __DEFAULT__ could not be found. [927a1e82-aa21-4358-b213-120682d85e63] 2021-01-20 13:05:32,833 - cinderlib-client - ERROR - Failure occurred when trying to run command 'create_volume': Volume type with name __DEFAULT__ could not be found. [b645c321-5c64-4c54-972a-f81080bb6b0f] 2021-01-20 15:16:45,667 - cinderlib-client - ERROR - Failure occurred when trying to run command 'connect_volume': Volume type with name __DEFAULT__ could not be found. [19d98e01] 2021-01-20 15:16:48,232 - cinderlib-client - ERROR - Failure occurred when trying to run command 'connect_volume': Volume type with name __DEFAULT__ could not be found. [2088beea] 2021-01-20 15:16:50,320 - cinderlib-client - ERROR - Failure occurred when trying to run command 'disconnect_volume': Volume type with name __DEFAULT__ could not be found. [793923ab] 2021-01-20 15:37:28,423 - cinderlib-client - ERROR - Failure occurred when trying to run command 'connect_volume': Volume type with name __DEFAULT__ could not be found. [5c8f0d73] 2021-01-20 15:37:30,707 - cinderlib-client - ERROR - Failure occurred when trying to run command 'connect_volume': Volume type with name __DEFAULT__ could not be found. [35e96dcd] 2021-01-20 15:55:36,179 - cinderlib-client - ERROR - Failure occurred when trying to run command 'create_volume': Volume type with name __DEFAULT__ could not be found. [b5fbb358-36c7-4d3b-9431-16b3a699f300] 2021-01-20 15:57:33,733 - cinderlib-client - ERROR - Failure occurred when trying to run command 'create_volume': Volume type with name __DEFAULT__ could not be found. [ae879975-d36a-4b2b-9829-2713266f1c1f] in engine.log: 021-01-20 15:57:31,795+01 INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task-2) [ae879975-d36a-4b2b-9829-2713266f1c1f] Running command: AddDiskCommand internal: false. Entities affected : ID: 32dd7b42-eeb7-4cf5-9bef-bd8f8dd9608e Type: StorageAction group CREATE_DISK with role type USER 2021-01-20 15:57:32,014+01 INFO [org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-1) [ae879975-d36a-4b2b-9829-2713266f1c1f] Running command: AddManagedBlockStorageDiskCommand internal: true. 2021-01-20 15:57:33,936+01 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedExecutorService-commandCoordinator-Thread-1) [ae879975-d36a-4b2b-9829-2713266f1c1f] cinderlib execution failed: 2021-01-20 15:57:34,012+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-2) [] EVENT_ID: USER_ADD_DISK_FINISHED_FAILURE(2,022), Add-Disk operation failed to complete. 2021-01-20 15:57:34,090+01 INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-47) [ae879975-d36a-4b2b-9829-2713266f1c1f] Command 'AddDisk' id: 'c7e43317-495e-4b97-96cd-02f02cb20ab2' child commands '[1de7bfa3-09c3-45a5-955a-580236f0296c]' executions were completed, status 'FAILED' 2021-01-20 15:57:35,127+01 ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48) [ae879975-d36a-4b2b-9829-2713266f1c1f] Ending command 'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' with failure. 2021-01-20 15:57:35,135+01 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48) [ae879975-d36a-4b2b-9829-2713266f1c1f] Ending command 'org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand' with failure. I check it on cinderlib 2.0, 2.1, 3.0 installed as from rpm (ussuri) and from pip install - errors the same. on backend we use CEPH 14 rbd pools
Can you submit a bug? Looks like a problem with `volume_types` is empty, can you run (assuming you use the default DB name/user): $ psql -U ovirt_cinderlib -d ovirt_cinderlib -c "\x on" -c "select * from volume_types" The output on my system is: -[ RECORD 1 ]+------------------------------------- created_at | 2020-08-05 08:55:20.408367 updated_at | 2020-08-05 08:55:20.408367 deleted_at | deleted | f id | fc7602c6-cb67-416e-a963-56f7f8f7eb42 name | __DEFAULT__ qos_specs_id | is_public | t description | Default Volume Type On Wed, Jan 20, 2021 at 5:13 PM Mike Andreev <mikhail.andreev@meduniwien.ac.at> wrote:
hi all, after upgrade oVirt 4.4.7 Cluster Compatibility version 4.4 ->4.5 cinderlib storage is died:
tail -f /var/log/ovirt-engine/cinderlib/cinderlib.log 2021-01-20 12:35:28,781 - cinderlib-client - ERROR - Failure occurred when trying to run command 'delete_volume': Volume type with name __DEFAULT__ could not be found. [3d4a9aa9-c5f0-498c-b650-bc64d979f194] 2021-01-20 12:43:31,209 - cinderlib-client - ERROR - Failure occurred when trying to run command 'delete_volume': Volume type with name __DEFAULT__ could not be found. [927a1e82-aa21-4358-b213-120682d85e63] 2021-01-20 13:05:32,833 - cinderlib-client - ERROR - Failure occurred when trying to run command 'create_volume': Volume type with name __DEFAULT__ could not be found. [b645c321-5c64-4c54-972a-f81080bb6b0f] 2021-01-20 15:16:45,667 - cinderlib-client - ERROR - Failure occurred when trying to run command 'connect_volume': Volume type with name __DEFAULT__ could not be found. [19d98e01] 2021-01-20 15:16:48,232 - cinderlib-client - ERROR - Failure occurred when trying to run command 'connect_volume': Volume type with name __DEFAULT__ could not be found. [2088beea] 2021-01-20 15:16:50,320 - cinderlib-client - ERROR - Failure occurred when trying to run command 'disconnect_volume': Volume type with name __DEFAULT__ could not be found. [793923ab] 2021-01-20 15:37:28,423 - cinderlib-client - ERROR - Failure occurred when trying to run command 'connect_volume': Volume type with name __DEFAULT__ could not be found. [5c8f0d73] 2021-01-20 15:37:30,707 - cinderlib-client - ERROR - Failure occurred when trying to run command 'connect_volume': Volume type with name __DEFAULT__ could not be found. [35e96dcd] 2021-01-20 15:55:36,179 - cinderlib-client - ERROR - Failure occurred when trying to run command 'create_volume': Volume type with name __DEFAULT__ could not be found. [b5fbb358-36c7-4d3b-9431-16b3a699f300] 2021-01-20 15:57:33,733 - cinderlib-client - ERROR - Failure occurred when trying to run command 'create_volume': Volume type with name __DEFAULT__ could not be found. [ae879975-d36a-4b2b-9829-2713266f1c1f]
in engine.log: 021-01-20 15:57:31,795+01 INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task-2) [ae879975-d36a-4b2b-9829-2713266f1c1f] Running command: AddDiskCommand internal: false. Entities affected : ID: 32dd7b42-eeb7-4cf5-9bef-bd8f8dd9608e Type: StorageAction group CREATE_DISK with role type USER 2021-01-20 15:57:32,014+01 INFO [org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-1) [ae879975-d36a-4b2b-9829-2713266f1c1f] Running command: AddManagedBlockStorageDiskCommand internal: true. 2021-01-20 15:57:33,936+01 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedExecutorService-commandCoordinator-Thread-1) [ae879975-d36a-4b2b-9829-2713266f1c1f] cinderlib execution failed: 2021-01-20 15:57:34,012+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-2) [] EVENT_ID: USER_ADD_DISK_FINISHED_FAILURE(2,022), Add-Disk operation failed to complete. 2021-01-20 15:57:34,090+01 INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-47) [ae879975-d36a-4b2b-9829-2713266f1c1f] Command 'AddDisk' id: 'c7e43317-495e-4b97-96cd-02f02cb20ab2' child commands '[1de7bfa3-09c3-45a5-955a-580236f0296c]' executions were completed, status 'FAILED' 2021-01-20 15:57:35,127+01 ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48) [ae879975-d36a-4b2b-9829-2713266f1c1f] Ending command 'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' with failure. 2021-01-20 15:57:35,135+01 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48) [ae879975-d36a-4b2b-9829-2713266f1c1f] Ending command 'org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand' with failure.
I check it on cinderlib 2.0, 2.1, 3.0 installed as from rpm (ussuri) and from pip install - errors the same. on backend we use CEPH 14 rbd pools _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RLZLTWMJE45VQJ...
I got only: [postgres@ovc-engine-44 ~]$ psql -U ovirt_cinderlib_20200824130037 -d ovirt_cinderlib_20200824130037 -c "\x on" -c "select * from volume_types" psql: error: FATAL: Peer authentication failed for user "ovirt_cinderlib_20200824130037"
cat /var/lib/pgsql/data/pg_hba.conf # TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all all peer host ovirt_engine_history ovirt_engine_history_grafana 0.0.0.0/0 md5 host ovirt_engine_history ovirt_engine_history_grafana ::0/0 md5 host ovirt_engine_history ovirt_engine_history 0.0.0.0/0 md5 host ovirt_engine_history ovirt_engine_history ::0/0 md5 host engine engine 0.0.0.0/0 md5 host engine engine ::0/0 md5 host ovirt_cinderlib_20200824130037 ovirt_cinderlib_20200824130037 0.0.0.0/0 md5 host ovirt_cinderlib_20200824130037 ovirt_cinderlib_20200824130037 ::0/0 md5 host ovirt_cinderlib ovirt_cinderlib 0.0.0.0/0 md5 host ovirt_cinderlib ovirt_cinderlib ::0/0 md5 # IPv4 local connections: host all all 127.0.0.1/32 ident # IPv6 local connections: host all all ::1/128 ident
looks like our main cinderlib DB is empty: sql -d ovirt_cinderlib_20200824130037 -c "\x on" -c "select * from volume_types" Expanded display is on. (0 rows)
I restore this table - "volume_types" from backup of DB and cinderlib work now, but I did not like how it happened just after changing the cluster compatibility level Restore command: pg_restore -d ovirt_cinderlib_20200824130037 -a -t volume_types /root/db/cinderlib_backup.db
participants (2)
-
Benny Zlotnik -
Mike Andreev