Hi guys, I'm in big troubles. Since last night my ovirt engine is unavailable (won't correctly start). Process ovirt-engine is running, and systemd doesn't complain (systemctl status is ok). Digging in engine.log I can find an error: 2022-07-13 09:57:48,314+02 ERROR [org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer] (ServerService Thread Pool -- 45) [] Cannot deserialize { "@class" : "org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters", "commandId" : [ "org.ovirt.engine.core.compat.Guid", { "uuid" : "6ae544f6-b608-4d8d-9f99-eabd5d5db0ad" } ], [...cut...] "domain" : "my.dom.ain"[truncated 5971 chars]; line: 72, column: 89] (through reference chain: org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters["diskImagesMap"]) 2022-07-13 09:57:48,315+02 ERROR [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 45) [] Failed to initialize backend: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap() on org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl@3f52ccce [...cut...] During the night there was a job that was moving a disk (2.5TB) from a storage domain to another. I think it's related. Any ideas? TIA, gc _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QK54KG2PMLJ...

Hi, engine version is ovirt-engine-4.5.1.3-1.el8.noarch, find attached relevant part of engine.log (from "systemctl start ovirt-engine" onward) and output from sql command. Thank you. Regards, gc On 13/07/2022 10:44, Benny Zlotnik wrote: > Which ovirt version is this? Can you attach the full log? > As well as the output of > $ psql -U engine -d engine -c "\x on" -c "select * from command_entities" > > This can likely be resolved by removing the relevant row from > command_entities, but first let's get a better understanding of what's > going on > > On Wed, Jul 13, 2022 at 11:40 AM Giulio Casella <giulio(a)di.unimi.it&gt; wrote: >> >> Hi guys, >> I'm in big troubles. >> >> Since last night my ovirt engine is unavailable (won't correctly start). >> Process ovirt-engine is running, and systemd doesn't complain (systemctl >> status is ok). >> Digging in engine.log I can find an error: >> >> 2022-07-13 09:57:48,314+02 ERROR >> [org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer] >> (ServerService Thread Pool -- 45) [] Cannot deserialize { >> "@class" : >> "org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters", >> "commandId" : [ "org.ovirt.engine.core.compat.Guid", { >> "uuid" : "6ae544f6-b608-4d8d-9f99-eabd5d5db0ad" >> } ], >> [...cut...] >> "domain" : "my.dom.ain"[truncated 5971 chars]; line: 72, column: 89] >> (through reference chain: >> org.ovirt.engine.core.common.action.CreateSnapshotDiskParameters["diskImagesMap"]) >> 2022-07-13 09:57:48,315+02 ERROR >> [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] >> (ServerService Thread Pool -- 45) [] Failed to initialize backend: >> org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke >> public void >> org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap() >> on org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl@3f52ccce >> [...cut...] >> >> During the night there was a job that was moving a disk (2.5TB) from a >> storage domain to another. I think it's related. >> >> Any ideas? >> >> >> TIA, >> gc >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QK54KG2PMLJ... >

Thanks Benny, On 13/07/2022 12:20, Benny Zlotnik wrote: > Thanks, is the operation complete? If yes, you can remove the rows > referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from > command_entities How can I know if the operation is completed? I've no access to admin portal anymore...

The only things I know are: - there are 2 command in command_entities referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad: the 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad itself and another one referencing it as a parent - both of them are marked as ENDED_SUCCESSFULLY in the status field - other commands in table exists, 3 of them marked as active (none of them references 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad)

On 13/07/2022 12:54, Benny Zlotnik wrote: > On Wed, Jul 13, 2022 at 1:48 PM Giulio Casella <giulio(a)di.unimi.it&gt; wrote: >> >> Thanks Benny, >> >> On 13/07/2022 12:20, Benny Zlotnik wrote: >>> Thanks, is the operation complete? If yes, you can remove the rows >>> referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from >>> command_entities >> >> How can I know if the operation is completed? I've no access to admin >> portal anymore... > Right, I forgot about this... > >> The only things I know are: >> - there are 2 command in command_entities referencing command_id >> 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad: the >> 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad itself and another one referencing >> it as a parent >> - both of them are marked as ENDED_SUCCESSFULLY in the status field >> - other commands in table exists, 3 of them marked as active (none of >> them references 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad) > > You can check in the job table, > > select * > from job > where correlation_id = 'd19c0b79-76dd-40df-af41-fc18e59bb536'; No job returned from job query. Can I proceed to remove 2 rows from command_entities referencing 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad? Thanks, gc

Done! After deleting some rows present in command_entities table (those that hadn't a job with a correlationId in job table) the engine restarted. I'll file a bugzilla for this issue. Thank you Benny. Ciao, gc On 13/07/2022 15:29, Benny Zlotnik wrote: > I will very carefully say yes, but please backup the engine before you do that > > On Wed, Jul 13, 2022 at 1:59 PM Giulio Casella <giulio(a)di.unimi.it&gt; wrote: >> >> >> >> On 13/07/2022 12:54, Benny Zlotnik wrote: >>> On Wed, Jul 13, 2022 at 1:48 PM Giulio Casella <giulio(a)di.unimi.it&gt; wrote: >>>> >>>> Thanks Benny, >>>> >>>> On 13/07/2022 12:20, Benny Zlotnik wrote: >>>>> Thanks, is the operation complete? If yes, you can remove the rows >>>>> referencing command_id 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad from >>>>> command_entities >>>> >>>> How can I know if the operation is completed? I've no access to admin >>>> portal anymore... >>> Right, I forgot about this... >>> >>>> The only things I know are: >>>> - there are 2 command in command_entities referencing command_id >>>> 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad: the >>>> 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad itself and another one referencing >>>> it as a parent >>>> - both of them are marked as ENDED_SUCCESSFULLY in the status field >>>> - other commands in table exists, 3 of them marked as active (none of >>>> them references 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad) >>> >>> You can check in the job table, >>> >>> select * >>> from job >>> where correlation_id = 'd19c0b79-76dd-40df-af41-fc18e59bb536'; >> >> No job returned from job query. >> Can I proceed to remove 2 rows from command_entities referencing >> 6ae544f6-b608-4d8d-9f99-eabd5d5db0ad? >> >> >> Thanks, >> gc >> >