1:17 p.m.
OK, then,
Please find the vdsm logs for both source and destination attached.
Eduardo Mayoral Jimeno (emayoral(a)arsys.es)
Administrador de sistemas. Departamento de Plataformas. Arsys internet.
+34 941 620 145 ext. 5153
On 16/03/17 12:13, Yaniv Kaul wrote:
Please share on the mailing list - I might not get to look at them.
It's not too big.
Y.
On Thu, Mar 16, 2017 at 1:06 PM, Eduardo Mayoral <emayoral(a)arsys.es
<mailto:emayoral@arsys.es>> wrote:
Sure! Please find the logs attached. I do not mind sharing them on
the mailing list, but I feel they are probably too big.
Eduardo Mayoral Jimeno (emayoral(a)arsys.es <mailto:emayoral@arsys.es>)
Administrador de sistemas. Departamento de Plataformas. Arsys internet.
+34 941 620 145 ext. 5153 <tel:+34%20941%2062%2001%2045>
On 16/03/17 11:58, Yaniv Kaul wrote:
>
>
> On Thu, Mar 16, 2017 at 12:56 PM, Eduardo Mayoral
> <emayoral(a)arsys.es <mailto:emayoral@arsys.es>> wrote:
>
> Hi,
>
> An interesting thing just happened on my oVirt deployment.
>
> While setting a host for maintenance, one of the VMs
> running on that
> host failed to migrate. Then ovirt-engine for some reason
> turned the VM
> off. Here are the relevant log lines from engine.log:
>
>
>
>
> 2017-03-16 09:56:23,324Z INFO
> [org.ovirt.engine.core.bll.MigrateVmCommand] (default task-60)
> [61a52216] Lock Acquired to object
>
'EngineLock:{exclusiveLocks='[422663a9-d712-4992-7c81-165b2976073e=<VM,
> ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED$VmName
> entorno127.arsysdesarrollo.lan>]', sharedLocks='null'}'
> 2017-03-16 09:56:23,957Z INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (default task-60) [61a52216] EVENT_ID:
> VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID:
> 61a52216, Job
> ID: fbc5e0d7-4618-4ca8-b7a7-0fd9a43f490f, Call Stack: null,
> Custom Event
> ID: -1, Message: Migration initiated by system (VM:
> entorno127.arsysdesarrollo.lan, Source: llkk594.arsyslan.es
> <http://llkk594.arsyslan.es>;,
> Destination: llkk593.arsyslan.es
> <http://llkk593.arsyslan.es>;, Reason: Host preparing for
> maintenance).
> 2017-03-16 09:56:26,818Z INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (DefaultQuartzScheduler10)
> [7c6c403b-0a82-47aa-aeb4-57fbe06e20e1] VM
>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
> was unexpectedly detected as 'MigratingTo' on VDS
> '43f43ec5-e51d-400d-8569-261c98382e3a'(llkk593.arsyslan.es
> <http://llkk593.arsyslan.es>) (expected on
> '11a82467-afa4-4e4e-bd92-3383082d0a5e')
> 2017-03-16 09:56:42,149Z INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (DefaultQuartzScheduler10)
> [fac8ade9-a867-4ff7-aac5-912f78cc3bb5] VM
>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
> was unexpectedly detected as 'MigratingTo' on VDS
> '43f43ec5-e51d-400d-8569-261c98382e3a'(llkk593.arsyslan.es
> <http://llkk593.arsyslan.es>) (expected on
> '11a82467-afa4-4e4e-bd92-3383082d0a5e')
> 2017-03-16 09:56:54,755Z INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (DefaultQuartzScheduler3)
> [54fa9a1b-fddb-4812-b5d2-06cf92834709] VM
>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
> moved from 'MigratingFrom' --> 'Down'
> 2017-03-16 09:56:54,755Z INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (DefaultQuartzScheduler3)
> [54fa9a1b-fddb-4812-b5d2-06cf92834709] Handing
> over VM
>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
> to Host '43f43ec5-e51d-400d-8569-261c98382e3a'. Setting VM to
> status
> 'MigratingTo'
> 2017-03-16 09:56:58,160Z INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (DefaultQuartzScheduler4)
> [dd4ea7bb-85bc-410e-9a8f-357c28d85d1c] VM
>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
> is running in db and not running on VDS
> '43f43ec5-e51d-400d-8569-261c98382e3a'(llkk593.arsyslan.es
> <http://llkk593.arsyslan.es>)
> 2017-03-16 09:56:58,161Z INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (DefaultQuartzScheduler4)
> [dd4ea7bb-85bc-410e-9a8f-357c28d85d1c] add VM
>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
> to rerun treatment
> turn to maintenance while VMs are still running on it.(VM:
> entorno127.arsysdesarrollo.lan, Source: llkk594.arsyslan.es
> <http://llkk594.arsyslan.es>;,
> Destination: llkk593.arsyslan.es <http://llkk593.arsyslan.es>).
> 2017-03-16 09:56:59,193Z INFO
> [org.ovirt.engine.core.bll.MigrateVmCommand]
> (org.ovirt.thread.pool-7-thread-27) [14d40a88] Lock freed to
> object
>
'EngineLock:{exclusiveLocks='[422663a9-d712-4992-7c81-165b2976073e=<VM,
> ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED$VmName
> entorno127.arsysdesarrollo.lan>]', sharedLocks='null'}'
>
> Now, I understand an VM migration may fail for a number of
> reasons, but
> in that case, shouldn't the VM keep running on the source
> host? I do not
> quite understand what happened here or how to avoid it in the
> future.
>
>
> Can you share vdsm logs from both source and destination?
> Y.
>
>
>
> Best regards,
>
> --
> Eduardo Mayoral Jimeno (emayoral(a)arsys.es
> <mailto:emayoral@arsys.es>)
> Administrador de sistemas. Departamento de Plataformas. Arsys
> internet.
> +34 941 620 145 ext. 5153
> <tel:%2B34%20941%20620%20145%20ext.%205153>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
> <http://lists.ovirt.org/mailman/listinfo/users>
>
>
9 p.m.
New subject: VM switched off when migration for host maintenance fails
Hi,
Is it reproducible?
It sounds similar to https://bugzilla.redhat.com/show_bug.cgi?id=1426727 but it’s not
exactly that
Can you confirm you’re not using QoS/Quota?
Does the VM fail the same way when you migrate it manually?
If not, can you please reproduce with Move to Maintenance flow and narrow it down -
timeframe, exact VM, corresponding engine and vdsm and qemu logs?
Thanks,
michal
On 16 Mar 2017, at 12:17, Eduardo Mayoral <emayoral(a)arsys.es>
wrote:
OK, then,
Please find the vdsm logs for both source and destination attached.
Eduardo Mayoral Jimeno (emayoral(a)arsys.es
)
Administrador de sistemas. Departamento de Plataformas. Arsys internet.
+34 941 620 145 ext. 5153
On 16/03/17 12:13, Yaniv Kaul wrote:
> Please share on the mailing list - I might not get to look at them.
> It's not too big.
> Y.
>
> On Thu, Mar 16, 2017 at 1:06 PM, Eduardo Mayoral <emayoral(a)arsys.es> wrote:
> Sure! Please find the logs attached. I do not mind sharing them on the mailing list,
but I feel they are probably too big.
> Eduardo Mayoral Jimeno (emayoral(a)arsys.es
> )
> Administrador de sistemas. Departamento de Plataformas. Arsys internet.
>
> +34 941 620 145 ext. 5153
> On 16/03/17 11:58, Yaniv Kaul wrote:
>>
>>
>> On Thu, Mar 16, 2017 at 12:56 PM, Eduardo Mayoral <emayoral(a)arsys.es>
wrote:
>> Hi,
>>
>> An interesting thing just happened on my oVirt deployment.
>>
>> While setting a host for maintenance, one of the VMs running on that
>> host failed to migrate. Then ovirt-engine for some reason turned the VM
>> off. Here are the relevant log lines from engine.log:
>>
>>
>>
>>
>> 2017-03-16 09:56:23,324Z INFO
>> [org.ovirt.engine.core.bll.MigrateVmCommand] (default task-60)
>> [61a52216] Lock Acquired to object
>>
'EngineLock:{exclusiveLocks='[422663a9-d712-4992-7c81-165b2976073e=<VM,
>> ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED$VmName
>> entorno127.arsysdesarrollo.lan>]', sharedLocks='null'}'
>> 2017-03-16 09:56:23,957Z INFO
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (default task-60) [61a52216] EVENT_ID:
>> VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 61a52216, Job
>> ID: fbc5e0d7-4618-4ca8-b7a7-0fd9a43f490f, Call Stack: null, Custom Event
>> ID: -1, Message: Migration initiated by system (VM:
>> entorno127.arsysdesarrollo.lan, Source: llkk594.arsyslan.es,
>> Destination: llkk593.arsyslan.es, Reason: Host preparing for maintenance).
>> 2017-03-16 09:56:26,818Z INFO
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>> (DefaultQuartzScheduler10) [7c6c403b-0a82-47aa-aeb4-57fbe06e20e1] VM
>> '422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>> was unexpectedly detected as 'MigratingTo' on VDS
>> '43f43ec5-e51d-400d-8569-261c98382e3a'(llkk593.arsyslan.es) (expected on
>> '11a82467-afa4-4e4e-bd92-3383082d0a5e')
>> 2017-03-16 09:56:42,149Z INFO
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>> (DefaultQuartzScheduler10) [fac8ade9-a867-4ff7-aac5-912f78cc3bb5] VM
>> '422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>> was unexpectedly detected as 'MigratingTo' on VDS
>> '43f43ec5-e51d-400d-8569-261c98382e3a'(llkk593.arsyslan.es) (expected on
>> '11a82467-afa4-4e4e-bd92-3383082d0a5e')
>> 2017-03-16 09:56:54,755Z INFO
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>> (DefaultQuartzScheduler3) [54fa9a1b-fddb-4812-b5d2-06cf92834709] VM
>> '422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>> moved from 'MigratingFrom' --> 'Down'
>> 2017-03-16 09:56:54,755Z INFO
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>> (DefaultQuartzScheduler3) [54fa9a1b-fddb-4812-b5d2-06cf92834709] Handing
>> over VM
>> '422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>> to Host '43f43ec5-e51d-400d-8569-261c98382e3a'. Setting VM to status
>> 'MigratingTo'
>> 2017-03-16 09:56:58,160Z INFO
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>> (DefaultQuartzScheduler4) [dd4ea7bb-85bc-410e-9a8f-357c28d85d1c] VM
>> '422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>> is running in db and not running on VDS
>> '43f43ec5-e51d-400d-8569-261c98382e3a'(llkk593.arsyslan.es)
>> 2017-03-16 09:56:58,161Z INFO
>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>> (DefaultQuartzScheduler4) [dd4ea7bb-85bc-410e-9a8f-357c28d85d1c] add VM
>> '422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>> to rerun treatment
>> turn to maintenance while VMs are still running on it.(VM:
>> entorno127.arsysdesarrollo.lan, Source: llkk594.arsyslan.es,
>> Destination: llkk593.arsyslan.es).
>> 2017-03-16 09:56:59,193Z INFO
>> [org.ovirt.engine.core.bll.MigrateVmCommand]
>> (org.ovirt.thread.pool-7-thread-27) [14d40a88] Lock freed to object
>>
'EngineLock:{exclusiveLocks='[422663a9-d712-4992-7c81-165b2976073e=<VM,
>> ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED$VmName
>> entorno127.arsysdesarrollo.lan>]', sharedLocks='null'}'
>>
>> Now, I understand an VM migration may fail for a number of reasons, but
>> in that case, shouldn't the VM keep running on the source host? I do not
>> quite understand what happened here or how to avoid it in the future.
>>
>> Can you share vdsm logs from both source and destination?
>> Y.
>>
>>
>> Best regards,
>>
>> --
>> Eduardo Mayoral Jimeno (emayoral(a)arsys.es)
>> Administrador de sistemas. Departamento de Plataformas. Arsys internet.
>> +34 941 620 145 ext. 5153
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
<vdsmlogs.tar.gz>_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
11:58 p.m.
New subject: VM switched off when migration for host maintenance fails
Hi, Michal, thank you for your interest.
Is it reproducible?
No. This is actually the first time that I have seen this behaviour, and a colleague of
mine is developing a script for automatic patching of the hosts, so we are putting them
into maintenance very frequently in the last few days.
It sounds similar to https://bugzilla.redhat.com/show_bug.cgi?id=1426727 but it’s not
exactly that
Can you confirm you’re not using QoS/Quota?
Yes, I can confirm we are not using QoS / Quota. Quota is disabled for the datacenter and
there are no QoS policies defined.
Does the VM fail the same way when you migrate it manually?
No, I just did it several times and it migrated just fine.
If not, can you please reproduce with Move to Maintenance flow and narrow it down -
timeframe, exact VM, corresponding engine and vdsm and qemu logs?
I will try to do this, but it will take some time, I will get back with the results next
week.
Best regards!
Eduardo Mayoral Jimeno (emayoral(a)arsys.es)
Administrador de sistemas. Departamento de Plataformas. Arsys internet.
+34 941 620 145 ext. 5153
On 16/03/17 20:00, Michal Skrivanek wrote:
Hi,
Is it reproducible?
It sounds similar to https://bugzilla.redhat.com/show_bug.cgi?id=1426727 but it’s not
exactly that
Can you confirm you’re not using QoS/Quota?
Does the VM fail the same way when you migrate it manually?
If not, can you please reproduce with Move to Maintenance flow and narrow it down -
timeframe, exact VM, corresponding engine and vdsm and qemu logs?
Thanks,
michal
> On 16 Mar 2017, at 12:17, Eduardo Mayoral <emayoral(a)arsys.es> wrote:
>
> OK, then,
>
> Please find the vdsm logs for both source and destination attached.
>
> Eduardo Mayoral Jimeno (emayoral(a)arsys.es
> )
> Administrador de sistemas. Departamento de Plataformas. Arsys internet.
> +34 941 620 145 ext. 5153
>
> On 16/03/17 12:13, Yaniv Kaul wrote:
>> Please share on the mailing list - I might not get to look at them.
>> It's not too big.
>> Y.
>>
>> On Thu, Mar 16, 2017 at 1:06 PM, Eduardo Mayoral <emayoral(a)arsys.es>
wrote:
>> Sure! Please find the logs attached. I do not mind sharing them on the mailing
list, but I feel they are probably too big.
>> Eduardo Mayoral Jimeno (emayoral(a)arsys.es
>> )
>> Administrador de sistemas. Departamento de Plataformas. Arsys internet.
>>
>> +34 941 620 145 ext. 5153
>> On 16/03/17 11:58, Yaniv Kaul wrote:
>>>
>>> On Thu, Mar 16, 2017 at 12:56 PM, Eduardo Mayoral <emayoral(a)arsys.es>
wrote:
>>> Hi,
>>>
>>> An interesting thing just happened on my oVirt deployment.
>>>
>>> While setting a host for maintenance, one of the VMs running on that
>>> host failed to migrate. Then ovirt-engine for some reason turned the VM
>>> off. Here are the relevant log lines from engine.log:
>>>
>>>
>>>
>>>
>>> 2017-03-16 09:56:23,324Z INFO
>>> [org.ovirt.engine.core.bll.MigrateVmCommand] (default task-60)
>>> [61a52216] Lock Acquired to object
>>>
'EngineLock:{exclusiveLocks='[422663a9-d712-4992-7c81-165b2976073e=<VM,
>>> ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED$VmName
>>> entorno127.arsysdesarrollo.lan>]', sharedLocks='null'}'
>>> 2017-03-16 09:56:23,957Z INFO
>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> (default task-60) [61a52216] EVENT_ID:
>>> VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 61a52216, Job
>>> ID: fbc5e0d7-4618-4ca8-b7a7-0fd9a43f490f, Call Stack: null, Custom Event
>>> ID: -1, Message: Migration initiated by system (VM:
>>> entorno127.arsysdesarrollo.lan, Source: llkk594.arsyslan.es,
>>> Destination: llkk593.arsyslan.es, Reason: Host preparing for maintenance).
>>> 2017-03-16 09:56:26,818Z INFO
>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>>> (DefaultQuartzScheduler10) [7c6c403b-0a82-47aa-aeb4-57fbe06e20e1] VM
>>>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>>> was unexpectedly detected as 'MigratingTo' on VDS
>>> '43f43ec5-e51d-400d-8569-261c98382e3a'(llkk593.arsyslan.es) (expected
on
>>> '11a82467-afa4-4e4e-bd92-3383082d0a5e')
>>> 2017-03-16 09:56:42,149Z INFO
>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>>> (DefaultQuartzScheduler10) [fac8ade9-a867-4ff7-aac5-912f78cc3bb5] VM
>>>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>>> was unexpectedly detected as 'MigratingTo' on VDS
>>> '43f43ec5-e51d-400d-8569-261c98382e3a'(llkk593.arsyslan.es) (expected
on
>>> '11a82467-afa4-4e4e-bd92-3383082d0a5e')
>>> 2017-03-16 09:56:54,755Z INFO
>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>>> (DefaultQuartzScheduler3) [54fa9a1b-fddb-4812-b5d2-06cf92834709] VM
>>>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>>> moved from 'MigratingFrom' --> 'Down'
>>> 2017-03-16 09:56:54,755Z INFO
>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>>> (DefaultQuartzScheduler3) [54fa9a1b-fddb-4812-b5d2-06cf92834709] Handing
>>> over VM
>>>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>>> to Host '43f43ec5-e51d-400d-8569-261c98382e3a'. Setting VM to status
>>> 'MigratingTo'
>>> 2017-03-16 09:56:58,160Z INFO
>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>>> (DefaultQuartzScheduler4) [dd4ea7bb-85bc-410e-9a8f-357c28d85d1c] VM
>>>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>>> is running in db and not running on VDS
>>> '43f43ec5-e51d-400d-8569-261c98382e3a'(llkk593.arsyslan.es)
>>> 2017-03-16 09:56:58,161Z INFO
>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>>> (DefaultQuartzScheduler4) [dd4ea7bb-85bc-410e-9a8f-357c28d85d1c] add VM
>>>
'422663a9-d712-4992-7c81-165b2976073e'(entorno127.arsysdesarrollo.lan)
>>> to rerun treatment
>>> turn to maintenance while VMs are still running on it.(VM:
>>> entorno127.arsysdesarrollo.lan, Source: llkk594.arsyslan.es,
>>> Destination: llkk593.arsyslan.es).
>>> 2017-03-16 09:56:59,193Z INFO
>>> [org.ovirt.engine.core.bll.MigrateVmCommand]
>>> (org.ovirt.thread.pool-7-thread-27) [14d40a88] Lock freed to object
>>>
'EngineLock:{exclusiveLocks='[422663a9-d712-4992-7c81-165b2976073e=<VM,
>>> ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED$VmName
>>> entorno127.arsysdesarrollo.lan>]', sharedLocks='null'}'
>>>
>>> Now, I understand an VM migration may fail for a number of reasons, but
>>> in that case, shouldn't the VM keep running on the source host? I do not
>>> quite understand what happened here or how to avoid it in the future.
>>>
>>> Can you share vdsm logs from both source and destination?
>>> Y.
>>>
>>>
>>> Best regards,
>>>
>>> --
>>> Eduardo Mayoral Jimeno (emayoral(a)arsys.es)
>>> Administrador de sistemas. Departamento de Plataformas. Arsys internet.
>>> +34 941 620 145 ext. 5153
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
> <vdsmlogs.tar.gz>_______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
2809
days inactive
2809
days old
2 comments
2 participants
participants (2)
-
Eduardo Mayoral -
Michal Skrivanek