10:12 a.m.
hi all,
after upgrade oVirt 4.4.7 Cluster Compatibility version 4.4 ->4.5 cinderlib storage is
died:
tail -f /var/log/ovirt-engine/cinderlib/cinderlib.log
2021-01-20 12:35:28,781 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'delete_volume': Volume type with name __DEFAULT__ could not be found.
[3d4a9aa9-c5f0-498c-b650-bc64d979f194]
2021-01-20 12:43:31,209 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'delete_volume': Volume type with name __DEFAULT__ could not be found.
[927a1e82-aa21-4358-b213-120682d85e63]
2021-01-20 13:05:32,833 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'create_volume': Volume type with name __DEFAULT__ could not be found.
[b645c321-5c64-4c54-972a-f81080bb6b0f]
2021-01-20 15:16:45,667 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'connect_volume': Volume type with name __DEFAULT__ could not be found.
[19d98e01]
2021-01-20 15:16:48,232 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'connect_volume': Volume type with name __DEFAULT__ could not be found.
[2088beea]
2021-01-20 15:16:50,320 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'disconnect_volume': Volume type with name __DEFAULT__ could not be found.
[793923ab]
2021-01-20 15:37:28,423 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'connect_volume': Volume type with name __DEFAULT__ could not be found.
[5c8f0d73]
2021-01-20 15:37:30,707 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'connect_volume': Volume type with name __DEFAULT__ could not be found.
[35e96dcd]
2021-01-20 15:55:36,179 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'create_volume': Volume type with name __DEFAULT__ could not be found.
[b5fbb358-36c7-4d3b-9431-16b3a699f300]
2021-01-20 15:57:33,733 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'create_volume': Volume type with name __DEFAULT__ could not be found.
[ae879975-d36a-4b2b-9829-2713266f1c1f]
in engine.log:
021-01-20 15:57:31,795+01 INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand]
(default task-2) [ae879975-d36a-4b2b-9829-2713266f1c1f] Running command: AddDiskCommand
internal: false. Entities affected : ID: 32dd7b42-eeb7-4cf5-9bef-bd8f8dd9608e Type:
StorageAction group CREATE_DISK with role type USER
2021-01-20 15:57:32,014+01 INFO
[org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand]
(EE-ManagedExecutorService-commandCoordinator-Thread-1)
[ae879975-d36a-4b2b-9829-2713266f1c1f] Running command: AddManagedBlockStorageDiskCommand
internal: true.
2021-01-20 15:57:33,936+01 ERROR
[org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor]
(EE-ManagedExecutorService-commandCoordinator-Thread-1)
[ae879975-d36a-4b2b-9829-2713266f1c1f] cinderlib execution failed:
2021-01-20 15:57:34,012+01 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-2) []
EVENT_ID: USER_ADD_DISK_FINISHED_FAILURE(2,022), Add-Disk operation failed to complete.
2021-01-20 15:57:34,090+01 INFO
[org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-47)
[ae879975-d36a-4b2b-9829-2713266f1c1f] Command 'AddDisk' id:
'c7e43317-495e-4b97-96cd-02f02cb20ab2' child commands
'[1de7bfa3-09c3-45a5-955a-580236f0296c]' executions were completed, status
'FAILED'
2021-01-20 15:57:35,127+01 ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48)
[ae879975-d36a-4b2b-9829-2713266f1c1f] Ending command
'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' with failure.
2021-01-20 15:57:35,135+01 ERROR
[org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48)
[ae879975-d36a-4b2b-9829-2713266f1c1f] Ending command
'org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand'
with failure.
I check it on cinderlib 2.0, 2.1, 3.0 installed as from rpm (ussuri) and from pip install
- errors the same.
on backend we use CEPH 14 rbd pools
10:25 a.m.
New subject: Cinderlib died after upgrade Cluster Compatibility version 4.4 ->4.5
Can you submit a bug?
Looks like a problem with `volume_types` is empty, can you run
(assuming you use the default DB name/user):
$ psql -U ovirt_cinderlib -d ovirt_cinderlib -c "\x on" -c "select *
from volume_types"
The output on my system is:
-[ RECORD 1 ]+-------------------------------------
created_at | 2020-08-05 08:55:20.408367
updated_at | 2020-08-05 08:55:20.408367
deleted_at |
deleted | f
id | fc7602c6-cb67-416e-a963-56f7f8f7eb42
name | __DEFAULT__
qos_specs_id |
is_public | t
description | Default Volume Type
On Wed, Jan 20, 2021 at 5:13 PM Mike Andreev
<mikhail.andreev(a)meduniwien.ac.at> wrote:
hi all,
after upgrade oVirt 4.4.7 Cluster Compatibility version 4.4 ->4.5 cinderlib storage is
died:
tail -f /var/log/ovirt-engine/cinderlib/cinderlib.log
2021-01-20 12:35:28,781 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'delete_volume': Volume type with name __DEFAULT__ could not be found.
[3d4a9aa9-c5f0-498c-b650-bc64d979f194]
2021-01-20 12:43:31,209 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'delete_volume': Volume type with name __DEFAULT__ could not be found.
[927a1e82-aa21-4358-b213-120682d85e63]
2021-01-20 13:05:32,833 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'create_volume': Volume type with name __DEFAULT__ could not be found.
[b645c321-5c64-4c54-972a-f81080bb6b0f]
2021-01-20 15:16:45,667 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'connect_volume': Volume type with name __DEFAULT__ could not be found.
[19d98e01]
2021-01-20 15:16:48,232 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'connect_volume': Volume type with name __DEFAULT__ could not be found.
[2088beea]
2021-01-20 15:16:50,320 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'disconnect_volume': Volume type with name __DEFAULT__ could not be found.
[793923ab]
2021-01-20 15:37:28,423 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'connect_volume': Volume type with name __DEFAULT__ could not be found.
[5c8f0d73]
2021-01-20 15:37:30,707 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'connect_volume': Volume type with name __DEFAULT__ could not be found.
[35e96dcd]
2021-01-20 15:55:36,179 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'create_volume': Volume type with name __DEFAULT__ could not be found.
[b5fbb358-36c7-4d3b-9431-16b3a699f300]
2021-01-20 15:57:33,733 - cinderlib-client - ERROR - Failure occurred when trying to run
command 'create_volume': Volume type with name __DEFAULT__ could not be found.
[ae879975-d36a-4b2b-9829-2713266f1c1f]
in engine.log:
021-01-20 15:57:31,795+01 INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand]
(default task-2) [ae879975-d36a-4b2b-9829-2713266f1c1f] Running command: AddDiskCommand
internal: false. Entities affected : ID: 32dd7b42-eeb7-4cf5-9bef-bd8f8dd9608e Type:
StorageAction group CREATE_DISK with role type USER
2021-01-20 15:57:32,014+01 INFO
[org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand]
(EE-ManagedExecutorService-commandCoordinator-Thread-1)
[ae879975-d36a-4b2b-9829-2713266f1c1f] Running command: AddManagedBlockStorageDiskCommand
internal: true.
2021-01-20 15:57:33,936+01 ERROR
[org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor]
(EE-ManagedExecutorService-commandCoordinator-Thread-1)
[ae879975-d36a-4b2b-9829-2713266f1c1f] cinderlib execution failed:
2021-01-20 15:57:34,012+01 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-2) []
EVENT_ID: USER_ADD_DISK_FINISHED_FAILURE(2,022), Add-Disk operation failed to complete.
2021-01-20 15:57:34,090+01 INFO
[org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-47)
[ae879975-d36a-4b2b-9829-2713266f1c1f] Command 'AddDisk' id:
'c7e43317-495e-4b97-96cd-02f02cb20ab2' child commands
'[1de7bfa3-09c3-45a5-955a-580236f0296c]' executions were completed, status
'FAILED'
2021-01-20 15:57:35,127+01 ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48)
[ae879975-d36a-4b2b-9829-2713266f1c1f] Ending command
'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' with failure.
2021-01-20 15:57:35,135+01 ERROR
[org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48)
[ae879975-d36a-4b2b-9829-2713266f1c1f] Ending command
'org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDiskCommand'
with failure.
I check it on cinderlib 2.0, 2.1, 3.0 installed as from rpm (ussuri) and from pip
install - errors the same.
on backend we use CEPH 14 rbd pools
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/RLZLTWMJE45...
4:20 p.m.
New subject: Cinderlib died after upgrade Cluster Compatibility version 4.4 ->4.5
I got only:
[postgres@ovc-engine-44 ~]$ psql -U ovirt_cinderlib_20200824130037 -d
ovirt_cinderlib_20200824130037 -c "\x on" -c "select * from
volume_types"
psql: error: FATAL: Peer authentication failed for user
"ovirt_cinderlib_20200824130037"
4:39 p.m.
New subject: Cinderlib died after upgrade Cluster Compatibility version 4.4 ->4.5
cat /var/lib/pgsql/data/pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all peer
host ovirt_engine_history ovirt_engine_history_grafana 0.0.0.0/0 md5
host ovirt_engine_history ovirt_engine_history_grafana ::0/0 md5
host ovirt_engine_history ovirt_engine_history 0.0.0.0/0 md5
host ovirt_engine_history ovirt_engine_history ::0/0 md5
host engine engine 0.0.0.0/0 md5
host engine engine ::0/0 md5
host ovirt_cinderlib_20200824130037 ovirt_cinderlib_20200824130037 0.0.0.0/0
md5
host ovirt_cinderlib_20200824130037 ovirt_cinderlib_20200824130037 ::0/0
md5
host ovirt_cinderlib ovirt_cinderlib 0.0.0.0/0 md5
host ovirt_cinderlib ovirt_cinderlib ::0/0 md5
# IPv4 local connections:
host all all 127.0.0.1/32 ident
# IPv6 local connections:
host all all ::1/128 ident
4:15 a.m.
New subject: Cinderlib died after upgrade Cluster Compatibility version 4.4 ->4.5
looks like our main cinderlib DB is empty:
sql -d ovirt_cinderlib_20200824130037 -c "\x on" -c "select * from
volume_types"
Expanded display is on.
(0 rows)
5:18 a.m.
New subject: Cinderlib died after upgrade Cluster Compatibility version 4.4 ->4.5
I restore this table - "volume_types" from backup of DB and cinderlib work now,
but I did not like how it happened just after changing the cluster compatibility level
Restore command:
pg_restore -d ovirt_cinderlib_20200824130037 -a -t volume_types
/root/db/cinderlib_backup.db
1205
days inactive
1206
days old
5 comments
2 participants
participants (2)
-
Benny Zlotnik -
Mike Andreev