5:34 p.m.
The "vdsm-tool dump-volume-chains" command on the iSCSI storage domain
shows one disk in "ILLEGAL" state while the gui shows 8 disk images in
the same state.
###########################################
# BEGIN COMMAND OUTPUT
###########################################
[root@KVM01 ~]# vdsm-tool dump-volume-chains 045c7fda-ab98-4905-876c-
00b5413a619f
Images volume chains (base volume first)
image: 477e73af-e7db-4914-81ed-89b3fbc876f7
- c8320522-f839-472e-9707-a75f6fbe5cb6
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: 882c73fc-a833-4e2e-8e6a-f714d80c0f0d
- 689220c0-70f8-475f-98b2-6059e735cd1f
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: 0ca8c49f-452e-4f61-a3fc-c4bf2711e200
- dac06a5c-c5a8-4f82-aa8d-5c7a382da0b3
status: OK, voltype: LEAF, format: RAW, legality: LEGAL,
type: PREALLOCATED
image: 0ca0b8f8-8802-46ae-a9f8-45d5647feeb7
- 51a6de7b-b505-4c46-ae2a-25fb9faad810
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: ae6d2c62-cfbb-4765-930f-c0a0e3bc07d0
- b2d39c7d-5b9b-498d-a955-0e99c9bd5f3c
status: OK, voltype: INTERNAL, format: COW, legality:
LEGAL, type: SPARSE
- bf962809-3de7-4264-8c68-6ac12d65c151
status: ILLEGAL, voltype: LEAF, format: COW, legality:
ILLEGAL, type: SPARSE
image: ff8c64c4-d52b-4812-b541-7f291f98d961
- 85f77cd5-2f86-49a9-a411-8539114d3035
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: 70fc19a2-75da-41bd-a1f6-eb857ed2f18f
- a8f27397-395f-4b62-93c4-52699f59ea4b
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: 2b315278-65f5-45e8-a51e-02b9bc84dcee
- a6e2150b-57fa-46eb-b205-017fe01b0e4b
status: OK, voltype: INTERNAL, format: COW, legality:
LEGAL, type: SPARSE
- 2d8e5c14-c923-49ac-8660-8e57b801e329
status: OK, voltype: INTERNAL, format: COW, legality:
LEGAL, type: SPARSE
- 43100548-b849-4762-bfc5-18a0f281df2e
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: bf4594b0-242e-4823-abfd-9398ce5e31b7
- 4608ce2e-f288-40da-b4e5-2a5e7f3bf837
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: 00efca9d-932a-45b3-92c3-80065c1a40ce
- a0bb00bc-cefa-4031-9b59-3cddc3a53a0a
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: 5ce704eb-3508-4c36-b0ce-444ebdd27e66
- e41f2c2d-0a79-49f1-8911-1535a82bd735
status: OK, voltype: LEAF, format: RAW, legality: LEGAL,
type: PREALLOCATED
image: 11288fa5-0019-4ac0-8a7d-1d455e5e1549
- 5df31efc-14dd-427c-b575-c0d81f47c6d8
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: a091f7df-5c64-4b6b-a806-f4bf3aad53bc
- 38138111-2724-44a4-bde1-1fd9d60a1f63
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: c0b302c4-4b9d-4759-bb80-de1e865ecd58
- d4db9ba7-1b39-4b48-b319-013ebc1d71ce
status: OK, voltype: LEAF, format: RAW, legality: LEGAL,
type: PREALLOCATED
image: 21123edb-f74f-440b-9c42-4c16ba06a2b7
- f3cc17aa-4336-4542-9ab0-9df27032be0b
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: ad486d26-4594-4d16-a402-68b45d82078a
- e87e0c7c-4f6f-45e9-90ca-cf34617da3f6
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: c30c7f11-7818-4592-97ca-9d5be46e2d8e
- cb53ad06-65e8-474d-94c3-9acf044d5a09
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: 998ac54a-0d91-431f-8929-fe62f5d7290a
- d11aa0ee-d793-4830-9120-3b118ca44b6c
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: a1e69838-0bdf-42f3-95a4-56e4084510a9
- f687c727-ec06-49f1-9762-b0195e0b549a
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: a29598fe-f94e-4215-8508-19ac24b082c8
- 29b9ff26-2386-4fb5-832e-b7129307ceb4
status: OK, voltype: LEAF, format: RAW, legality: LEGAL,
type: PREALLOCATED
image: b151d4d7-d7fc-43ff-8bb2-75cf947ed626
- 34676d55-695a-4d2a-a7fa-546971067829
status: OK, voltype: LEAF, format: COW, legality: LEGAL,
type: SPARSE
image: 352a3a9a-4e1a-41bf-af86-717e374a7562
- adcc7655-9586-48c1-90d2-1dc9a851bbe1
status: OK, voltype: LEAF, format: RAW, legality: LEGAL,
type: PREALLOCATED
###########################################
# END COMMAND OUTPUT
########
###################################
###########################################
# BEGIN LOG OUTPUT FROM ENGINE
###########################################
The below output is an excerpt from the engine.log while attemtping to start one of the
afflicted VM's. Following the storage chain out i have discovered that
"919d6991-43e4-4f26-868e-031a01011191" does not exist. This is likely due to a
failure of the backup python script that the client is using. I can provide that script if
you all would like.
2016-04-20 08:56:58,285
INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-8) [] Correlation ID: abd1342, Job ID:
1ad2ee48-2c2c-437e-997b-469e09498e41, Call Stack: null, Custom Event ID: -1, Message: VM
Bill-V was started by admin@internal (Host: KVM03).
2016-04-20 08:57:00,392 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-1) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1,
Message: VM Bill-V is down with error. Exit message: Unable to get volume size for domain
045c7fda-ab98-4905-876c- 00b5413a619f volume 919d6991-43e4-4f26-868e-031a01011191.
2016-04-20 08:57:00,393 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(ForkJoinPool-1-worker-1) [] VM '6ef30172-b010-46fa-9482-accd30682232(Bill-V) is
running in db and not running in VDS 'KVM03'
2016-04-20 08:57:00,498
WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-5) [] Correlation ID: abd1342, Job ID:
1ad2ee48-2c2c-437e-997b-469e09498e41, Call Stack: null, Custom Event ID: -1, Message:
Failed to run VM Bill-V on Host KVM03.
###########################################
# END LOG OUTPUT FROM ENGINE
###########################################
I have followed the storage chain out to where the UUID'ed snapshots live, and
discovered that all of the "ILLEGAL" snapshots show to be broken symbolic
links.
Attached is a screenshot of the snapshots as they appear in the GUI. ALL of the UUID's
illustrated show as broken symbolic links in the storage domains.
On Tue, 2016-04-19 at 21:28 +0300, Nir Soffer wrote:
On Mon, Apr 18, 2016 at 3:16 PM, Clint Boggio
<clint(a)theboggios.com>
wrote:
>
> OVirt 3.6, 4 node cluster with dedicated engine. Main storage
> domain is iscsi, ISO and Export domains are NFS.
>
> Several of my VM snapshot disks show to be in an "illegal state".
> The system will not allow me to manipulate the snapshots in any
> way, nor clone the active system, or create a new snapshot.
>
> In the logs I see that the system complains about not being able to
> "get volume size for xxx", and also that the system appears to
> believe that the image is "locked" and is currently in the snapshot
> process.
>
> Of the VM's with this status, one rebooted and was lost due to
> "cannot get volume size for domain xxx".
>
Can you share the vdsm log showing these errors?
Also it may helpful to get the output of this command:
vdsm-tool dump-volume-chains SDUUID
Nir
>
> I fear that in this current condition, should any of the other
> machine reboot, they too will be lost.
>
> How can I troubleshoot this problem further, and hopefully
> alleviate the condition ?
>
> Thank you for your help.
>
> Clint
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
