6:11 a.m.
Hi. After upgrading from 3.4 to 3.5 (I've followed the official RHEV documentation)
the Hosted Engine VM cannot boot up anymore.
"hosted-engine --vm-status" says "Engine status: unknown stale-data"
agent.log says: Error: ''NoneType' object has no attribute
'iteritems''
Some logs:
- agent on node ov0h21: http://fpaste.org/144822/
- agent on node ov0h21: http://fpaste.org/144824/
- "hosted-engine --vm-status": http://fpaste.org/144825/
Thank you,
--
Stefano Stagnaro
Prisma Engineering S.r.l.
Via Petrocchi, 4
20127 Milano – Italy
Tel. 02 26113507 int 339
e-mail: stefanos(a)prisma-eng.com
skype: stefano.stagnaro
6:30 a.m.
On 10/24/2014 01:11 PM, Stefano Stagnaro wrote:
Hi. After upgrading from 3.4 to 3.5 (I've followed the official
RHEV documentation) the Hosted Engine VM cannot boot up anymore.
"hosted-engine --vm-status" says "Engine status: unknown stale-data"
agent.log says: Error: ''NoneType' object has no attribute
'iteritems''
Some logs:
- agent on node ov0h21: http://fpaste.org/144822/
- agent on node ov0h21: http://fpaste.org/144824/
- "hosted-engine --vm-status": http://fpaste.org/144825/
Hi Stefano,
can you please provide also the broker log?
Thank you,
Jirka
Thank you,
6:45 a.m.
On 10/24/2014 01:30 PM, Jiri Moskovcak wrote:
On 10/24/2014 01:11 PM, Stefano Stagnaro wrote:
> Hi. After upgrading from 3.4 to 3.5 (I've followed the official RHEV
> documentation) the Hosted Engine VM cannot boot up anymore.
>
> "hosted-engine --vm-status" says "Engine status: unknown
stale-data"
> agent.log says: Error: ''NoneType' object has no attribute
'iteritems''
>
> Some logs:
> - agent on node ov0h21: http://fpaste.org/144822/
> - agent on node ov0h21: http://fpaste.org/144824/
> - "hosted-engine --vm-status": http://fpaste.org/144825/
Hi Stefano,
can you please provide also the broker log?
- and also this file:
/rhev/data-center/mnt/ov0nfs:_engine/e4e8282e-6bde-4332-ad68-313287b4fc65/ha_agent/hosted-engine.metadata
Thanks,
Jirka
Thank you,
Jirka
>
> Thank you,
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
7:12 a.m.
Hi Jirka,
thank you for the reply. I've uploaded all the relevant logs in here:
https://www.dropbox.com/sh/qh2rbews45ky2g8/AAC4_4_j94cw6sI_hfaSFg-Fa?dl=0
Thank you,
--
Stefano Stagnaro
Prisma Engineering S.r.l.
Via Petrocchi, 4
20127 Milano – Italy
Tel. 02 26113507 int 339
e-mail: stefanos(a)prisma-eng.com
skype: stefano.stagnaro
On Fri, 2014-10-24 at 13:30 +0200, Jiri Moskovcak wrote:
On 10/24/2014 01:11 PM, Stefano Stagnaro wrote:
> Hi. After upgrading from 3.4 to 3.5 (I've followed the official RHEV
documentation) the Hosted Engine VM cannot boot up anymore.
>
> "hosted-engine --vm-status" says "Engine status: unknown
stale-data"
> agent.log says: Error: ''NoneType' object has no attribute
'iteritems''
>
> Some logs:
> - agent on node ov0h21: http://fpaste.org/144822/
> - agent on node ov0h21: http://fpaste.org/144824/
> - "hosted-engine --vm-status": http://fpaste.org/144825/
Hi Stefano,
can you please provide also the broker log?
Thank you,
Jirka
>
> Thank you,
>
8 a.m.
On 10/24/2014 02:12 PM, Stefano Stagnaro wrote:
Hi Jirka,
thank you for the reply. I've uploaded all the relevant logs in here:
https://www.dropbox.com/sh/qh2rbews45ky2g8/AAC4_4_j94cw6sI_hfaSFg-Fa?dl=0
Thank you,
Hi Stefano,
I'd say, that agent is not able to parse the metadata from the previous
version, so as a workaround before I fix it you can try to zero out the
metadata file (backup the original just in case..)
1. stop agent and broker on all hosts
2. truncate the file
this should do the trick:
$ service ovirt-ha-agent stop; service ovirt-ha-broker stop;
$ truncate --size 0
/rhev/data-center/mnt/ov0nfs:_engine/e4e8282e-6bde-4332-ad68-313287b4fc65/ha_agent/hosted-engine.metadata
$ truncate --size 1M
/rhev/data-center/mnt/ov0nfs:_engine/e4e8282e-6bde-4332-ad68-313287b4fc65/ha_agent/hosted-engine.metadata
$ service ovirt-ha-broker start; service ovirt-ha-agent start
--Jirka
12:07 p.m.
Hi Jirka,
after truncating the metadata file the Engine is running again.
Unfortunately now the HA migration is not working anymore. If I put the host with the
running Engine in local maintenance, the HA go trough EngineMigratingAway ->
ReinitializeFSM -> LocalMaintenance but the VM never migrates.
I can read this error in the agent.log:
MainThread::ERROR::2014-10-27
18:02:51,053::hosted_engine::867::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitor_migration)
Failed to migrate
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 863, in _monitor_migration
vm_id,
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/vds_client.py",
line 85, in run_vds_client_cmd
response['status']['message'])
DetailedError: Error 12 from migrateStatus: Fatal error during migration
Thank you,
--
Stefano Stagnaro
Prisma Engineering S.r.l.
Via Petrocchi, 4
20127 Milano – Italy
Tel. 02 26113507 int 339
e-mail: stefanos(a)prisma-eng.com
skype: stefano.stagnaro
On Fri, 2014-10-24 at 15:00 +0200, Jiri Moskovcak wrote:
On 10/24/2014 02:12 PM, Stefano Stagnaro wrote:
> Hi Jirka,
>
> thank you for the reply. I've uploaded all the relevant logs in here:
https://www.dropbox.com/sh/qh2rbews45ky2g8/AAC4_4_j94cw6sI_hfaSFg-Fa?dl=0
>
> Thank you,
>
Hi Stefano,
I'd say, that agent is not able to parse the metadata from the previous
version, so as a workaround before I fix it you can try to zero out the
metadata file (backup the original just in case..)
1. stop agent and broker on all hosts
2. truncate the file
this should do the trick:
$ service ovirt-ha-agent stop; service ovirt-ha-broker stop;
$ truncate --size 0
/rhev/data-center/mnt/ov0nfs:_engine/e4e8282e-6bde-4332-ad68-313287b4fc65/ha_agent/hosted-engine.metadata
$ truncate --size 1M
/rhev/data-center/mnt/ov0nfs:_engine/e4e8282e-6bde-4332-ad68-313287b4fc65/ha_agent/hosted-engine.metadata
$ service ovirt-ha-broker start; service ovirt-ha-agent start
--Jirka
2:19 a.m.
On 10/27/2014 06:07 PM, Stefano Stagnaro wrote:
Hi Jirka,
after truncating the metadata file the Engine is running again.
Unfortunately now the HA migration is not working anymore. If I put the host with the
running Engine in local maintenance, the HA go trough EngineMigratingAway ->
ReinitializeFSM -> LocalMaintenance but the VM never migrates.
I can read this error in the agent.log:
MainThread::ERROR::2014-10-27
18:02:51,053::hosted_engine::867::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitor_migration)
Failed to migrate
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 863, in _monitor_migration
vm_id,
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/vds_client.py",
line 85, in run_vds_client_cmd
response['status']['message'])
DetailedError: Error 12 from migrateStatus: Fatal error during migration
Thank you,
Hi,
to debug the migration failure I gonna need the engine.log.
Thanks,
Jirka
5:32 a.m.
Hi,
please find the new logs at the same place:
https://www.dropbox.com/sh/qh2rbews45ky2g8/AAC4_4_j94cw6sI_hfaSFg-Fa?dl=0
Thank you,
--
Stefano Stagnaro
Prisma Engineering S.r.l.
Via Petrocchi, 4
20127 Milano – Italy
Tel. 02 26113507 int 339
e-mail: stefanos(a)prisma-eng.com
skype: stefano.stagnaro
On Wed, 2014-10-29 at 08:19 +0100, Jiri Moskovcak wrote:
On 10/27/2014 06:07 PM, Stefano Stagnaro wrote:
> Hi Jirka,
>
> after truncating the metadata file the Engine is running again.
>
> Unfortunately now the HA migration is not working anymore. If I put the host with
the running Engine in local maintenance, the HA go trough EngineMigratingAway ->
ReinitializeFSM -> LocalMaintenance but the VM never migrates.
>
> I can read this error in the agent.log:
>
> MainThread::ERROR::2014-10-27
18:02:51,053::hosted_engine::867::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitor_migration)
Failed to migrate
> Traceback (most recent call last):
> File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 863, in _monitor_migration
> vm_id,
> File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/vds_client.py",
line 85, in run_vds_client_cmd
> response['status']['message'])
> DetailedError: Error 12 from migrateStatus: Fatal error during migration
>
> Thank you,
>
Hi,
to debug the migration failure I gonna need the engine.log.
Thanks,
Jirka
1:49 a.m.
On 10/29/2014 11:32 AM, Stefano Stagnaro wrote:
Hi,
please find the new logs at the same place:
https://www.dropbox.com/sh/qh2rbews45ky2g8/AAC4_4_j94cw6sI_hfaSFg-Fa?dl=0
Thank you,
Hi,
I can't see any exception related to the migration process, but it's
full of some exception about getting storage data from database. Nir,
can you please take a look if it's something critical which might
influence the migration process?
Thanks,
Jirka
8:41 a.m.
On 10/24/2014 02:12 PM, Stefano Stagnaro wrote:
Hi Jirka,
thank you for the reply. I've uploaded all the relevant logs in here:
https://www.dropbox.com/sh/qh2rbews45ky2g8/AAC4_4_j94cw6sI_hfaSFg-Fa?dl=0
Thank you,
After closer look I noticed, that there is 'null' as the engine-state in
the metadata. This shouldn't happen, but we also should be able to
recover when this happens.
--Jirka
3488
days inactive
3494
days old
9 comments
2 participants
participants (2)
-
Jiri Moskovcak -
Stefano Stagnaro