Cannot deserialize - engine doesn't start
Hi guys, I'm in big troubles. Since last night my ovirt engine is unavailable (won't correctly start). Process ovirt-engine is running, and systemd doesn't complain (systemctl status is ok). Digging in engine.log I can find an error: 2022-07-13 09:57:48,314+02 ERROR [org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer] (ServerService Thread Pool -- 45) [] Cannot deserialize { "@class" : "org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters", "commandId" : [ "org.ovirt.engine.core.compat.Guid", { "uuid" : "6ae544f6-b608-4d8d-9f99-eabd5d5db0ad" } ], [...cut...] "domain" : "my.dom.ain"[truncated 5971 chars]; line: 72, column: 89] (through reference chain: org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters["diskImagesMap"]) 2022-07-13 09:57:48,315+02 ERROR [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 45) [] Failed to initialize backend: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap() on org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl@3f52ccce [...cut...] During the night there was a job that was moving a disk (2.5TB) from a storage domain to another. I think it's related. Any ideas? TIA, gc
Which ovirt version is this? Can you attach the full log? As well as the output of $ psql -U engine -d engine -c "\x on" -c "select * from command_entities" This can likely be resolved by removing the relevant row from command_entities, but first let's get a better understanding of what's going on On Wed, Jul 13, 2022 at 11:40 AM Giulio Casella <giulio@di.unimi.it> wrote:
Hi guys, I'm in big troubles.
Since last night my ovirt engine is unavailable (won't correctly start). Process ovirt-engine is running, and systemd doesn't complain (systemctl status is ok). Digging in engine.log I can find an error:
2022-07-13 09:57:48,314+02 ERROR [org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer] (ServerService Thread Pool -- 45) [] Cannot deserialize { "@class" : "org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters", "commandId" : [ "org.ovirt.engine.core.compat.Guid", { "uuid" : "6ae544f6-b608-4d8d-9f99-eabd5d5db0ad" } ], [...cut...] "domain" : "my.dom.ain"[truncated 5971 chars]; line: 72, column: 89] (through reference chain: org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters["diskImagesMap"]) 2022-07-13 09:57:48,315+02 ERROR [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 45) [] Failed to initialize backend: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap() on org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl@3f52ccce [...cut...]
During the night there was a job that was moving a disk (2.5TB) from a storage domain to another. I think it's related.
Any ideas?
TIA, gc _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QK54KG2PMLJQVG...
Hi, engine version is ovirt-engine-4.5.1.3-1.el8.noarch, find attached relevant part of engine.log (from "systemctl start ovirt-engine" onward) and output from sql command. Thank you. Regards, gc On 13/07/2022 10:44, Benny Zlotnik wrote:
Which ovirt version is this? Can you attach the full log? As well as the output of $ psql -U engine -d engine -c "\x on" -c "select * from command_entities"
This can likely be resolved by removing the relevant row from command_entities, but first let's get a better understanding of what's going on
On Wed, Jul 13, 2022 at 11:40 AM Giulio Casella <giulio@di.unimi.it> wrote:
Hi guys, I'm in big troubles.
Since last night my ovirt engine is unavailable (won't correctly start). Process ovirt-engine is running, and systemd doesn't complain (systemctl status is ok). Digging in engine.log I can find an error:
2022-07-13 09:57:48,314+02 ERROR [org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer] (ServerService Thread Pool -- 45) [] Cannot deserialize { "@class" : "org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters", "commandId" : [ "org.ovirt.engine.core.compat.Guid", { "uuid" : "6ae544f6-b608-4d8d-9f99-eabd5d5db0ad" } ], [...cut...] "domain" : "my.dom.ain"[truncated 5971 chars]; line: 72, column: 89] (through reference chain: org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters["diskImagesMap"]) 2022-07-13 09:57:48,315+02 ERROR [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 45) [] Failed to initialize backend: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap() on org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl@3f52ccce [...cut...]
During the night there was a job that was moving a disk (2.5TB) from a storage domain to another. I think it's related.
Any ideas?
TIA, gc _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QK54KG2PMLJQVG...
Thanks, is the operation complete? If yes, you can remove the rows referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from command_entities Also, please submit a bug for this On Wed, Jul 13, 2022 at 12:04 PM Giulio Casella <giulio@di.unimi.it> wrote:
Hi, engine version is ovirt-engine-4.5.1.3-1.el8.noarch, find attached relevant part of engine.log (from "systemctl start ovirt-engine" onward) and output from sql command.
Thank you.
Regards, gc
On 13/07/2022 10:44, Benny Zlotnik wrote:
Which ovirt version is this? Can you attach the full log? As well as the output of $ psql -U engine -d engine -c "\x on" -c "select * from command_entities"
This can likely be resolved by removing the relevant row from command_entities, but first let's get a better understanding of what's going on
On Wed, Jul 13, 2022 at 11:40 AM Giulio Casella <giulio@di.unimi.it> wrote:
Hi guys, I'm in big troubles.
Since last night my ovirt engine is unavailable (won't correctly start). Process ovirt-engine is running, and systemd doesn't complain (systemctl status is ok). Digging in engine.log I can find an error:
2022-07-13 09:57:48,314+02 ERROR [org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer] (ServerService Thread Pool -- 45) [] Cannot deserialize { "@class" : "org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters", "commandId" : [ "org.ovirt.engine.core.compat.Guid", { "uuid" : "6ae544f6-b608-4d8d-9f99-eabd5d5db0ad" } ], [...cut...] "domain" : "my.dom.ain"[truncated 5971 chars]; line: 72, column: 89] (through reference chain: org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters["diskImagesMap"]) 2022-07-13 09:57:48,315+02 ERROR [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 45) [] Failed to initialize backend: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap() on org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl@3f52ccce [...cut...]
During the night there was a job that was moving a disk (2.5TB) from a storage domain to another. I think it's related.
Any ideas?
TIA, gc _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QK54KG2PMLJQVG...
Thanks Benny, On 13/07/2022 12:20, Benny Zlotnik wrote:
Thanks, is the operation complete? If yes, you can remove the rows referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from command_entities
How can I know if the operation is completed? I've no access to admin portal anymore... The only things I know are: - there are 2 command in command_entities referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad: the 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad itself and another one referencing it as a parent - both of them are marked as ENDED_SUCCESSFULLY in the status field - other commands in table exists, 3 of them marked as active (none of them references 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad)
Also, please submit a bug for this
Ok, I will.
On Wed, Jul 13, 2022 at 1:48 PM Giulio Casella <giulio@di.unimi.it> wrote:
Thanks Benny,
On 13/07/2022 12:20, Benny Zlotnik wrote:
Thanks, is the operation complete? If yes, you can remove the rows referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from command_entities
How can I know if the operation is completed? I've no access to admin portal anymore...
Right, I forgot about this...
The only things I know are: - there are 2 command in command_entities referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad: the 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad itself and another one referencing it as a parent - both of them are marked as ENDED_SUCCESSFULLY in the status field - other commands in table exists, 3 of them marked as active (none of them references 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad)
You can check in the job table, select * from job where correlation_id = 'd19c0b79-76dd-40df-af41-fc18e59bb536';
On 13/07/2022 12:54, Benny Zlotnik wrote:
On Wed, Jul 13, 2022 at 1:48 PM Giulio Casella <giulio@di.unimi.it> wrote:
Thanks Benny,
On 13/07/2022 12:20, Benny Zlotnik wrote:
Thanks, is the operation complete? If yes, you can remove the rows referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from command_entities
How can I know if the operation is completed? I've no access to admin portal anymore...
Right, I forgot about this...
The only things I know are: - there are 2 command in command_entities referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad: the 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad itself and another one referencing it as a parent - both of them are marked as ENDED_SUCCESSFULLY in the status field - other commands in table exists, 3 of them marked as active (none of them references 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad)
You can check in the job table,
select * from job where correlation_id = 'd19c0b79-76dd-40df-af41-fc18e59bb536';
No job returned from job query. Can I proceed to remove 2 rows from command_entities referencing 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad? Thanks, gc
I will very carefully say yes, but please backup the engine before you do that On Wed, Jul 13, 2022 at 1:59 PM Giulio Casella <giulio@di.unimi.it> wrote:
On 13/07/2022 12:54, Benny Zlotnik wrote:
On Wed, Jul 13, 2022 at 1:48 PM Giulio Casella <giulio@di.unimi.it> wrote:
Thanks Benny,
On 13/07/2022 12:20, Benny Zlotnik wrote:
Thanks, is the operation complete? If yes, you can remove the rows referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from command_entities
How can I know if the operation is completed? I've no access to admin portal anymore...
Right, I forgot about this...
The only things I know are: - there are 2 command in command_entities referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad: the 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad itself and another one referencing it as a parent - both of them are marked as ENDED_SUCCESSFULLY in the status field - other commands in table exists, 3 of them marked as active (none of them references 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad)
You can check in the job table,
select * from job where correlation_id = 'd19c0b79-76dd-40df-af41-fc18e59bb536';
No job returned from job query. Can I proceed to remove 2 rows from command_entities referencing 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad?
Thanks, gc
Done! After deleting some rows present in command_entities table (those that hadn't a job with a correlationId in job table) the engine restarted. I'll file a bugzilla for this issue. Thank you Benny. Ciao, gc On 13/07/2022 15:29, Benny Zlotnik wrote:
I will very carefully say yes, but please backup the engine before you do that
On Wed, Jul 13, 2022 at 1:59 PM Giulio Casella <giulio@di.unimi.it> wrote:
On 13/07/2022 12:54, Benny Zlotnik wrote:
On Wed, Jul 13, 2022 at 1:48 PM Giulio Casella <giulio@di.unimi.it> wrote:
Thanks Benny,
On 13/07/2022 12:20, Benny Zlotnik wrote:
Thanks, is the operation complete? If yes, you can remove the rows referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from command_entities
How can I know if the operation is completed? I've no access to admin portal anymore...
Right, I forgot about this...
The only things I know are: - there are 2 command in command_entities referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad: the 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad itself and another one referencing it as a parent - both of them are marked as ENDED_SUCCESSFULLY in the status field - other commands in table exists, 3 of them marked as active (none of them references 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad)
You can check in the job table,
select * from job where correlation_id = 'd19c0b79-76dd-40df-af41-fc18e59bb536';
No job returned from job query. Can I proceed to remove 2 rows from command_entities referencing 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad?
Thanks, gc
Thanks, I posted a fix[1], please share the bug link after you submit it and I'll attach the PR to it [1] https://github.com/oVirt/ovirt-engine/pull/526 On Wed, Jul 13, 2022 at 5:26 PM Giulio Casella <giulio@di.unimi.it> wrote:
Done! After deleting some rows present in command_entities table (those that hadn't a job with a correlationId in job table) the engine restarted.
I'll file a bugzilla for this issue.
Thank you Benny.
Ciao, gc
On 13/07/2022 15:29, Benny Zlotnik wrote:
I will very carefully say yes, but please backup the engine before you do that
On Wed, Jul 13, 2022 at 1:59 PM Giulio Casella <giulio@di.unimi.it> wrote:
On 13/07/2022 12:54, Benny Zlotnik wrote:
On Wed, Jul 13, 2022 at 1:48 PM Giulio Casella <giulio@di.unimi.it> wrote:
Thanks Benny,
On 13/07/2022 12:20, Benny Zlotnik wrote:
Thanks, is the operation complete? If yes, you can remove the rows referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from command_entities
How can I know if the operation is completed? I've no access to admin portal anymore...
Right, I forgot about this...
The only things I know are: - there are 2 command in command_entities referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad: the 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad itself and another one referencing it as a parent - both of them are marked as ENDED_SUCCESSFULLY in the status field - other commands in table exists, 3 of them marked as active (none of them references 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad)
You can check in the job table,
select * from job where correlation_id = 'd19c0b79-76dd-40df-af41-fc18e59bb536';
No job returned from job query. Can I proceed to remove 2 rows from command_entities referencing 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad?
Thanks, gc
Hi Benny, here the bug link: https://bugzilla.redhat.com/show_bug.cgi?id=2107985 Thank you, gc On 13/07/2022 16:48, Benny Zlotnik wrote:
Thanks, I posted a fix[1], please share the bug link after you submit it and I'll attach the PR to it
[1] https://github.com/oVirt/ovirt-engine/pull/526
On Wed, Jul 13, 2022 at 5:26 PM Giulio Casella <giulio@di.unimi.it> wrote:
Done! After deleting some rows present in command_entities table (those that hadn't a job with a correlationId in job table) the engine restarted.
I'll file a bugzilla for this issue.
Thank you Benny.
Ciao, gc
On 13/07/2022 15:29, Benny Zlotnik wrote:
I will very carefully say yes, but please backup the engine before you do that
On Wed, Jul 13, 2022 at 1:59 PM Giulio Casella <giulio@di.unimi.it> wrote:
On 13/07/2022 12:54, Benny Zlotnik wrote:
On Wed, Jul 13, 2022 at 1:48 PM Giulio Casella <giulio@di.unimi.it> wrote:
Thanks Benny,
On 13/07/2022 12:20, Benny Zlotnik wrote: > Thanks, is the operation complete? If yes, you can remove the rows > referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from > command_entities
How can I know if the operation is completed? I've no access to admin portal anymore...
Right, I forgot about this...
The only things I know are: - there are 2 command in command_entities referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad: the 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad itself and another one referencing it as a parent - both of them are marked as ENDED_SUCCESSFULLY in the status field - other commands in table exists, 3 of them marked as active (none of them references 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad)
You can check in the job table,
select * from job where correlation_id = 'd19c0b79-76dd-40df-af41-fc18e59bb536';
No job returned from job query. Can I proceed to remove 2 rows from command_entities referencing 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad?
Thanks, gc
participants (2)
-
Benny Zlotnik -
Giulio Casella