[ovirt-users] Cannot login after upgrade from 3.5 to 3.6

Sun Jan 31 09:31:09 UTC 2016


----- Original Message -----
> From: "Amador Pahim" <apahim at redhat.com>
> To: "Eli Mesika" <emesika at redhat.com>, "Marcelo Leandro" <marceloltmm at gmail.com>
> Cc: "Martin Perina" <mperina at redhat.com>, "Moti Asayag" <masayag at redhat.com>, "Allon Mureinik" <amureini at redhat.com>,
> "users" <users at ovirt.org>
> Sent: Thursday, January 28, 2016 6:18:33 PM
> Subject: Re: [ovirt-users] Cannot login after upgrade from 3.5 to 3.6
> 
> On 01/28/2016 01:59 PM, Eli Mesika wrote:
> >
> > ----- Original Message -----
> >> From: "Marcelo Leandro" <marceloltmm at gmail.com>
> >> To: "Amador Pahim" <apahim at redhat.com>
> >> Cc: "Eli Mesika" <emesika at redhat.com>, "Martin Perina"
> >> <mperina at redhat.com>, "Moti Asayag" <masayag at redhat.com>,
> >> "Allon Mureinik" <amureini at redhat.com>, "users" <users at ovirt.org>
> >> Sent: Thursday, January 28, 2016 1:39:49 PM
> >> Subject: Re: [ovirt-users] Cannot login after upgrade from 3.5 to 3.6
> >>
> >> Hello,
> >> Yes i can.
> >>
> >> It's here:
> >> https://copy.com/OmnU13SAPXh13ZiR
> > Yes
> >
> > The problem is in the command_entities table that is not cleaned up and has
> > two records after taskcleaner.sh is invoked by engine-setup
> > Since engine tries to deserialize classes defined in this table and as
> > Martin noted changes between 3.5 and 3.6 changed some classes, we got this
> > exception
> > We should not allow leftovers to exists in this table during the upgrade
> > process and taskcleaner utility should handle that as well
> > Please open a BZ on that, I will handle that
> > Meanwhile, as a workaround, run the following before the upgrade
> >
> > psql -U engine -c "DELETE from command_entities;" <database name>
> 
> I'm afraid the upgrade was done already. Will this work after the
> upgrade? Or he's supposed to downgrade/restore env to 3.5, run the
> DELETE and then try to upgrade again? Asking 'cause It's quite a task if
> yes :)

No need to downgrade, just run the query and retry 

> 
> >
> >
> >
> >> Thanks.
> >>
> >> 2016-01-28 8:17 GMT-03:00 Amador Pahim <apahim at redhat.com>:
> >>> On 01/28/2016 09:10 AM, Eli Mesika wrote:
> >>>>
> >>>> ----- Original Message -----
> >>>>> From: "Martin Perina" <mperina at redhat.com>
> >>>>> To: "Eli Mesika" <emesika at redhat.com>
> >>>>> Cc: "Moti Asayag" <masayag at redhat.com>, "Allon Mureinik"
> >>>>> <amureini at redhat.com>, "users" <users at ovirt.org>, "Amador
> >>>>> Pahim" <apahim at redhat.com>, "Marcelo Leandro" <marceloltmm at gmail.com>
> >>>>> Sent: Thursday, January 28, 2016 12:59:02 PM
> >>>>> Subject: Re: [ovirt-users] Cannot login after upgrade from 3.5 to 3.6
> >>>>>
> >>>>>
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> From: "Eli Mesika" <emesika at redhat.com>
> >>>>>> To: "Martin Perina" <mperina at redhat.com>
> >>>>>> Cc: "Moti Asayag" <masayag at redhat.com>, "Allon Mureinik"
> >>>>>> <amureini at redhat.com>, "users" <users at ovirt.org>, "Amador
> >>>>>> Pahim" <apahim at redhat.com>, "Marcelo Leandro" <marceloltmm at gmail.com>
> >>>>>> Sent: Thursday, January 28, 2016 11:55:56 AM
> >>>>>> Subject: Re: [ovirt-users] Cannot login after upgrade from 3.5 to 3.6
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ----- Original Message -----
> >>>>>>> From: "Martin Perina" <mperina at redhat.com>
> >>>>>>> To: "Moti Asayag" <masayag at redhat.com>, "Allon Mureinik"
> >>>>>>> <amureini at redhat.com>
> >>>>>>> Cc: "Eli Mesika" <emesika at redhat.com>, "users" <users at ovirt.org>,
> >>>>>>> "Amador
> >>>>>>> Pahim" <apahim at redhat.com>, "Marcelo
> >>>>>>> Leandro" <marceloltmm at gmail.com>
> >>>>>>> Sent: Wednesday, January 27, 2016 2:31:20 PM
> >>>>>>> Subject: Re: [ovirt-users] Cannot login after upgrade from 3.5 to 3.6
> >>>>>>>
> >>>>>>> So now I finally understand the issue:
> >>>>>>>
> >>>>>>> When you try to login, CommandBase tries to execute LoginCommand, but
> >>>>>>> before the command execution it loads content of commands cache and
> >>>>>>> here comes the issue: there's stored AddVmFromSnapshotCommand which
> >>>>>>> contains DiskImage as a parameter and DiskImage implementation has
> >>>>>>> changed between 3.5 and 3.6.
> >>>>>>>
> >>>>>>> Moti, is there some way how to clean up commands cache so it doesn't
> >>>>>>> contain those 3.5 commands?
> >>>>>>>
> >>>>>>> Also should those situation be covered during engine upgrades?
> >>>>>> Martin, is this a left over in the command_entities table that caused
> >>>>>> the
> >>>>>> failure while the entity is deserialized ?
> >>>>> Probably, because this DiskImage.Id serialization exception started to
> >>>>> appear
> >>>>> after upgrade from 3.5. Is there any way how to verify and clean up?
> >>>> I suggest having a bug on the taskcleaner code which is called upon
> >>>> upgrade, then I will try to reproduce
> >>>> Amador, Is there a chance we can get a backup of that 3.5 DB in order to
> >>>> reproduce ?
> >>>
> >>> Yes, I was talking to Marcelo off-list and he has the dump taken by
> >>> engine-setup. Marcelo, could you send the tar.gz'ed 3.5 dump download
> >>> link
> >>> off-list to Eli?
> >>> Thank you.
> >>>
> >>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>> Martin
> >>>>>>>
> >>>>>>>
> >>>>>>> ----- Original Message -----
> >>>>>>>> From: "Marcelo Leandro" <marceloltmm at gmail.com>
> >>>>>>>> To: "Martin Perina" <mperina at redhat.com>
> >>>>>>>> Cc: "Eli Mesika" <emesika at redhat.com>, "users" <users at ovirt.org>,
> >>>>>>>> "Amador
> >>>>>>>> Pahim" <apahim at redhat.com>
> >>>>>>>> Sent: Wednesday, January 27, 2016 1:08:19 PM
> >>>>>>>> Subject: Re: [ovirt-users] Cannot login after upgrade from 3.5 to
> >>>>>>>> 3.6
> >>>>>>>>
> >>>>>>>> The server  just in time  isolate from the network,is it a problem?
> >>>>>>>> thats ok?
> >>>>>>>>
> >>>>>>>> follows the logs:
> >>>>>>>>
> >>>>>>>> server.log:
> >>>>>>>> https://copy.com/Eni9adZSCjyMZNFN
> >>>>>>>>
> >>>>>>>> engine.log:
> >>>>>>>> https://copy.com/ayczW9ipHdW4BWni
> >>>>>>>>
> >>>>>>>> 2016-01-27 5:32 GMT-03:00 Martin Perina <mperina at redhat.com>:
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> according to the test of login by ovirt-engine-extensions-tool you
> >>>>>>>>> were
> >>>>>>>>> logged in
> >>>>>>>>> successfully, so there shouldn't be any issues logging into engine
> >>>>>>>>> using
> >>>>>>>>> admin at internal ...
> >>>>>>>>>
> >>>>>>>>> Unfortunately both engine.log and server.log ends on Jan 22th and
> >>>>>>>>> I'd
> >>>>>>>>> really
> >>>>>>>>> like to see up-to-date logs.
> >>>>>>>>>
> >>>>>>>>> So if you still cannot login into webadmin please do following:
> >>>>>>>>>
> >>>>>>>>> 1. Stop engine service
> >>>>>>>>>
> >>>>>>>>>        systemctl stop ovirt-engine
> >>>>>>>>>
> >>>>>>>>>      or
> >>>>>>>>>
> >>>>>>>>>        service ovirt-engine stop
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2. Rotate logs
> >>>>>>>>>
> >>>>>>>>>      cd /var/log/ovirt-engine
> >>>>>>>>>      mv engine.log engine-20160127.log
> >>>>>>>>>      mv server.log server-20160127.log
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 3. Start engine
> >>>>>>>>>
> >>>>>>>>>        systemctl start ovirt-engine
> >>>>>>>>>
> >>>>>>>>>      or
> >>>>>>>>>
> >>>>>>>>>        service ovirt-engine start
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 4. Try to login as admin with domain internal using webadmin
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 5. Try to login as admin at internal using REST API by
> >>>>>>>>>      accessing
> >>>>>>>>>
> >>>>>>>>>        http://SERVER/ovirt-engine/api/datacenters
> >>>>>>>>>
> >>>>>>>>>      Into user please enter full username: admin at internal
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> When done please attach latest server.log and engine.log
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks a lot
> >>>>>>>>>
> >>>>>>>>> Martin Perina
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ----- Original Message -----
> >>>>>>>>>> From: "Marcelo Leandro" <marceloltmm at gmail.com>
> >>>>>>>>>> To: "Martin Perina" <mperina at redhat.com>
> >>>>>>>>>> Cc: "Eli Mesika" <emesika at redhat.com>, "users" <users at ovirt.org>,
> >>>>>>>>>> "Amador
> >>>>>>>>>> Pahim" <apahim at redhat.com>
> >>>>>>>>>> Sent: Tuesday, January 26, 2016 8:46:12 PM
> >>>>>>>>>> Subject: Re: [ovirt-users] Cannot login after upgrade from 3.5 to
> >>>>>>>>>> 3.6
> >>>>>>>>>>
> >>>>>>>>>> hello,
> >>>>>>>>>>
> >>>>>>>>>> setup.log:
> >>>>>>>>>> https://copy.com/lIYzccOnwCiW7e9q
> >>>>>>>>>>
> >>>>>>>>>> and output the command :
> >>>>>>>>>> https://copy.com/7psvpyUzUq0FKW31
> >>>>>>>>>>
> >>>>>>>>>> error screen when i try to login on webadmin:
> >>>>>>>>>> https://copy.com/CfKZePnGFmIwF7B8
> >>>>>>>>>>
> >>>>>>>>>> Thanks
> >>>>>>>>>>
> >>>>>>>>>> 2016-01-26 12:24 GMT-03:00 Martin Perina <mperina at redhat.com>:
> >>>>>>>>>>> And also could you try to login using extension test tool:
> >>>>>>>>>>>
> >>>>>>>>>>>     ovirt-engine-extensions-tool aaa login-user
> >>>>>>>>>>>     --profile=internal
> >>>>>>>>>>>     --user-name=admin --password=interactive:
> >>>>>>>>>>>
> >>>>>>>>>>> and post the output of the command.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks a lot
> >>>>>>>>>>>
> >>>>>>>>>>> Martin Perina
> >>>>>>>>>>>
> >>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>> From: "Martin Perina" <mperina at redhat.com>
> >>>>>>>>>>>> To: "Marcelo Leandro" <marceloltmm at gmail.com>
> >>>>>>>>>>>> Cc: "Eli Mesika" <emesika at redhat.com>, "users"
> >>>>>>>>>>>> <users at ovirt.org>,
> >>>>>>>>>>>> "Amador
> >>>>>>>>>>>> Pahim" <apahim at redhat.com>
> >>>>>>>>>>>> Sent: Tuesday, January 26, 2016 4:12:21 PM
> >>>>>>>>>>>> Subject: Re: [ovirt-users] Cannot login after upgrade from 3.5
> >>>>>>>>>>>> to
> >>>>>>>>>>>> 3.6
> >>>>>>>>>>>>
> >>>>>>>>>>>> Could you please share also your server.log?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks
> >>>>>>>>>>>>
> >>>>>>>>>>>> Martin Perina
> >>>>>>>>>>>>
> >>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>> From: "Marcelo Leandro" <marceloltmm at gmail.com>
> >>>>>>>>>>>>> To: "Martin Perina" <mperina at redhat.com>
> >>>>>>>>>>>>> Cc: "Eli Mesika" <emesika at redhat.com>, "users"
> >>>>>>>>>>>>> <users at ovirt.org>,
> >>>>>>>>>>>>> "Amador
> >>>>>>>>>>>>> Pahim" <apahim at redhat.com>
> >>>>>>>>>>>>> Sent: Tuesday, January 26, 2016 4:03:16 PM
> >>>>>>>>>>>>> Subject: Re: [ovirt-users] Cannot login after upgrade from 3.5
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>> 3.6
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> And here the engine.log:
> >>>>>>>>>>>>> https://copy.com/TWjgIu7KwSkcmRZ8
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> the log started 2016-01-19.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2016-01-26 11:57 GMT-03:00 Marcelo Leandro
> >>>>>>>>>>>>> <marceloltmm at gmail.com>:
> >>>>>>>>>>>>>> I didn't change nothing on jan 20th and it happened on 22th.
> >>>>>>>>>>>>>> I can't login into the web admin.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2016-01-26 9:55 GMT-03:00 Martin Perina <mperina at redhat.com>:
> >>>>>>>>>>>>>>> Hi Marcelo,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> sorry for late response, too many tasks on me at the moment
> >>>>>>>>>>>>>>> :-(
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> As Eli said important issues started at 2016-01-20, but
> >>>>>>>>>>>>>>> engine-setup
> >>>>>>>>>>>>>>> log you posted is dated at 2016-01-22. So, did you perform
> >>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>> upgrade on important configuration changed on Jan 20th? If
> >>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>> you please share also this setup log?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regarding login: I haven't found any issue logging, I see
> >>>>>>>>>>>>>>> many
> >>>>>>>>>>>>>>> successful
> >>>>>>>>>>>>>>> logins of admin at internal in the log. So are you able to
> >>>>>>>>>>>>>>> login
> >>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>> webadmin
> >>>>>>>>>>>>>>> or not?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Martin Perina
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>> From: "Marcelo Leandro" <marceloltmm at gmail.com>
> >>>>>>>>>>>>>>>> To: "Eli Mesika" <emesika at redhat.com>
> >>>>>>>>>>>>>>>> Cc: "Martin Perina" <mperina at redhat.com>, "users"
> >>>>>>>>>>>>>>>> <users at ovirt.org>,
> >>>>>>>>>>>>>>>> "Amador Pahim" <apahim at redhat.com>
> >>>>>>>>>>>>>>>> Sent: Tuesday, January 26, 2016 12:27:55 PM
> >>>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Cannot login after upgrade from
> >>>>>>>>>>>>>>>> 3.5
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> 3.6
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Martins, can you help me ? I can't administer my
> >>>>>>>>>>>>>>>> environment.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 2016-01-25 8:30 GMT-03:00 Eli Mesika <emesika at redhat.com>:
> >>>>>>>>>>>>>>>>> Adding Martin
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> There are some aaa errors before that, but first
> >>>>>>>>>>>>>>>>> significant
> >>>>>>>>>>>>>>>>> error
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> 2016-01-20 00:14:55,002
> >>>>>>>>>>>>>>>>> Can you take a look ?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>>>> From: "Marcelo Leandro" <marceloltmm at gmail.com>
> >>>>>>>>>>>>>>>>>> To: "Eli Mesika" <emesika at redhat.com>
> >>>>>>>>>>>>>>>>>> Cc: users at ovirt.org, "Amador Pahim" <apahim at redhat.com>
> >>>>>>>>>>>>>>>>>> Sent: Sunday, January 24, 2016 1:48:04 PM
> >>>>>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Cannot login after upgrade
> >>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>> 3.5
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> 3.6
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> It's here : https://copy.com/TWjgIu7KwSkcmRZ8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>>>>> From: "Amador Pahim" <apahim at redhat.com>
> >>>>>>>>>>>>>>>>>>> To: "Marcelo Leandro" <marceloltmm at gmail.com>,
> >>>>>>>>>>>>>>>>>>> users at ovirt.org
> >>>>>>>>>>>>>>>>>>> Cc: emesika at redhat.com
> >>>>>>>>>>>>>>>>>>> Sent: Friday, January 22, 2016 7:08:20 PM
> >>>>>>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Cannot login after upgrade
> >>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>> 3.5
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> 3.6
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 01/22/2016 03:03 PM, Marcelo Leandro wrote:
> >>>>>>>>>>>>>>>>>>>> After engine-setup from
> >>>>>>>>>>>>>>>>>>>> ovirt-engine-3.5.6.2-1.el7.centos
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> ovirt-engine-3.6.1.3-1.el7.centos, I'm not able to
> >>>>>>>>>>>>>>>>>>>> login
> >>>>>>>>>>>>>>>>>>>> anymore.
> >>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>> engine.log say
> >>>>>>>>>>>>>>>>>>>> Caused by:
> >>>>>>>>>>>>>>>>>>>> org.codehaus.jackson.map.JsonMappingException:
> >>>>>>>>>>>>>>>>>>>> Invalid
> >>>>>>>>>>>>>>>>>>>> type
> >>>>>>>>>>>>>>>>>>>> id
> >>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.core.common.businessentities.DiskImage'
> >>>>>>>>>>>>>>>>>>>> (for
> >>>>>>>>>>>>>>>>>>>> id
> >>>>>>>>>>>>>>>>>>>> type 'Id.class'): no such class found (through
> >>>>>>>>>>>>>>>>>>>> reference
> >>>>>>>>>>>>>>>>>>>> chain:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.common.action.AddVmFromSnapshotParameters["vm"]->org.ovirt.engine.core.common.businessentities.VM["diskList"])
> >>>>>>>>>>>>>>>>>>> hmm, I see a similar msg here:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/pipermail/devel/2015-July/010976.html
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Not sure if related. Eli? Any thoughts here?
> >>>>>>>>>>>>>>>>>> I don't think it is related ...
> >>>>>>>>>>>>>>>>>> Amador, can you please attach a full engine log ?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Here the setup log:
> >>>>>>>>>>>>>>>>>>>> https://copy.com/YBddV8bg3tnjHnwp
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> And here the engine.log:
> >>>>>>>>>>>>>>>>>>>> https://copy.com/TWjgIu7KwSkcmRZ8
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Need urgent assistance. Thank you in advance.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>> Marcelo Leandro
> >>>>>>>>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>>>>>>>> Users mailing list
> >>>>>>>>>>>>>>>>>>>> Users at ovirt.org
> >>>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
> >>>>>>>>>>>>>>>>>>>
> 
>