9 a.m.
Hi Ala,
Yes, there's no way to remove it. I actually found a way to workaround
the issue, I'm posting it so other people can have a clue how to solve
it...
I'm fully aware that touching the DB directly is not a good idea but
I've been unable to find an alternative way.
1) Find the broken snapshot in the snapshots table and delete it.
engine=# select snapshot_id,snapshot_type,status,description from
snapshots where vm_id='343db85c-64bc-4f0c-b9a0-4ca8d129e0c3';
snapshot_id | snapshot_type | status |
description
--------------------------------------+---------------+--------+-------------------------------------------
3d1eaf0a-49b3-45be-a104-f5ceebe52540 | ACTIVE | OK | Active
VM
cb8672bb-38d3-47ee-a498-4b403fc7d8db | REGULAR | OK | Broken
snapshot
(2 rows)
2) Find the image linked to the broken snapshot (you must find the disk
in the Disks tab and write the UUID).
engine=# select
image_guid,parentid,imagestatus,vm_snapshot_id,volume_type,volume_format,active
from images where image_group_id='6cf2c490-784b-437f-8305-1bed40dc9c9d';
image_guid | parentid
| imagestatus | vm_snapshot_id | volume_type |
volume_format | active
--------------------------------------+--------------------------------------+-------------+--------------------------------------+-------------+---------------+--------
b7af66ad-d27b-4087-9c33-11625912a45f |
00000000-0000-0000-0000-000000000000 | 4 |
cb8672bb-38d3-47ee-a498-4b403fc7d8db | 1 | 5 | f
7f14ae53-feac-4088-9560-c77a16dcd5e3 |
b7af66ad-d27b-4087-9c33-11625912a45f | 1 |
3d1eaf0a-49b3-45be-a104-f5ceebe52540 | 2 | 4 | t
(2 rows)
3) Delete the broken snapshot from the snapshots table.
engine=# delete from snapshots where
snapshot_id='cb8672bb-38d3-47ee-a498-4b403fc7d8db';
DELETE 1
4) Delete the associated image to the broken snapshots.
engine=# delete from images where
image_guid='7f14ae53-feac-4088-9560-c77a16dcd5e3';
DELETE 1
At this time, the snapshot is no longer shown on the 'Snapshots' tab of
the VM. However, when starting the VM, I get an error with something
like this:
VM SED-tpl is down with error. Exit message: Bad volume specification
{u'index': 0, u'domainID':
u'110ea376-d789-40a1-b9f6-6b40c31afe01',
'reqsize': '0', u'format': u'cow', u'bootOrder':
u'1', u'address':
{u'function': u'0x0', u'bus': u'0x00', u'domain':
u'0x0000', u'type':
u'pci', u'slot': u'0x06'}, u'volumeID':
u'538600a5-31ab-40af-b326-d56bfc92bb0b', 'apparentsize':
'34359738368',
u'imageID': u'e05874d2-fb8a-4fd2-94ff-2f4bc6438d47', u'discard':
False,
u'specParams': {}, u'readonly': u'false', u'iface':
u'virtio',
u'optional': u'false', u'deviceId':
u'e05874d2-fb8a-4fd2-94ff-2f4bc6438d47', 'truesize':
'34359738368',
u'poolID': u'75bf8f48-970f-42bc-8596-f8ab6efb2b63', u'device':
u'disk',
u'shared': u'false', u'propagateErrors': u'off',
u'type': u'disk'}.
Now, get the storage domain ID on which the failed snapshot is. You can
know that by looking at the 'storage_domains' table in the DB. Then, run
the command that Benny mentioned (being the last UUID the one of the
storage domain):
vdsm-tool -vvv dump-volume-chains
bc0480e2-85fe-42a4-91ae-f733b23c801f
That will provide a map of the image list with all volumes and their
statuses. You should see at least an ILLEGAL one. Despite removing the
entries from the DB, they seem to still show up because the snapshot is
still in metadata and needs to be set as LEGAL. To set it as legal,
start the VM on a specific host where previously you are 'tail -f'ing
the vdsm.log. You'll see an entry like this:
2018-06-19 12:13:26,832+0100 INFO (vm/5bf9a0bb) [vdsm.api] START
prepareImage(sdUUID=u'bc0480e2-85fe-42a4-91ae-f733b23c801f',
spUUID=u'75bf8f48-970f-42bc-8596-f8ab6efb2b63',
imgUUID=u'870f7e85-d9b6-494a-9541-b419fb0e1b32',
leafUUID=u'd7fa8c51-8cad-4695-b90a-a8d1dc146371', allowIllegal=False)
from=internal, task_id=89770233-103d-47f3-acc1-45d2e96d9e91 (api:46)
Now, go to the SPM and run a command like this:
vdsClient -s yourhost.com setVolumeLegality
bc0480e2-85fe-42a4-91ae-f733b23c801f
75bf8f48-970f-42bc-8596-f8ab6efb2b63
870f7e85-d9b6-494a-9541-b419fb0e1b32
d7fa8c51-8cad-4695-b90a-a8d1dc146371 LEGAL
The VM is now able to power up. I know this is not a clean solution as
this leaves orphaned snapshots on storage domains, but up until now
we've not been able to find a better solution. At least now we know
machines can be powered up and no data loss happened.
If you have any additional tips we'd be glad to know so we can apply
them.
Thanks.
El 2018-06-19 14:41, Ala Hino escribió:
> Hi,
>
> Did you try to remove the same snapshot while the VM is down?
>
> On Tue, Jun 19, 2018 at 10:44 AM, <nicolas(a)devels.es> wrote:
>
>> Hi Benny,
>>
>> I used the tool to track one of the illegal volumes:
>>
>> image: e05874d2-fb8a-4fd2-94ff-2f4bc6438d47
>>
>> [...]
>>
>> - 887f486b-15cf-4083-9b35-8b7821a7841a
>> status: ILLEGAL, voltype: LEAF, format: COW,
>> legality: ILLEGAL, type: SPARSE
>>
>> So I tracked 887f486b-15cf-4083-9b35-8b7821a7841a in the logs and I
>> saw:
>>
>> 2018-06-16 04:46:20,818+01 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand]
>> (pool-5-thread-3) [cfc392ec-dc9f-418d-8156-d05c8e7ab9f8] START,
>> GetVolumeInfoVDSCommand(HostName = host.domain.es [1],
>>
>
GetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]',
>> runAsync='true', hostId='b2dfb945-d767-44aa-a547-2d1a4381f8e3',
>> storagePoolId='75bf8f48-970f-42bc-8596-f8ab6efb2b63',
>> storageDomainId='110ea376-d789-40a1-b9f6-6b40c31afe01',
>> imageGroupId='e05874d2-fb8a-4fd2-94ff-2f4bc6438d47',
>> imageId='887f486b-15cf-4083-9b35-8b7821a7841a'}), log id: 2a795424
>>
>> 2018-06-16 04:46:22,256+01 ERROR
>> [org.ovirt.engine.core.bll.DestroyImageCheckCommand]
>> (pool-5-thread-3) [cfc392ec-dc9f-418d-8156-d05c8e7ab9f8] The
>> following images were not removed:
>> [887f486b-15cf-4083-9b35-8b7821a7841a]
>>
>> 2018-06-16 04:47:44,900+01 ERROR
>>
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand]
>> (DefaultQuartzScheduler10) [cfc392ec-dc9f-418d-8156-d05c8e7ab9f8]
>> Snapshot '7b6f43ac-d3ad-47b2-8882-f5dccd74cf07' images
>>
>
'887f486b-15cf-4083-9b35-8b7821a7841a'..'538600a5-31ab-40af-b326-d56bfc92bb0b'
>> merged, but volume removal failed. Some or all of the following
>> volumes may be orphaned: [887f486b-15cf-4083-9b35-8b7821a7841a].
>> Please retry Live Merge on the snapshot to complete the operation.
>>
>> Can you provide some additional steps?
>>
>> Thank you!
>>
>> El 2018-06-18 18:27, Benny Zlotnik escribió:
>>
>> We prevent starting VMs with illegal images[1]
>>
>> You can use "$ vdsm-tool dump-volume-chains"
>> to look for illegal images and then look in the engine log for the
>> reason they became illagal,
>>
>> if it's something like this, it usually means you can remove them:
>>
>> 63696:2018-06-15 09:41:58,134+01 ERROR
>>
> [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand]
>> (DefaultQuartzScheduler2) [6fa97ea4-8f61-4a48-8e08-a8bb1b9de826]
>> Merging of snapshot 'e609d6cc-2025-4cf0-ad34-03519131cdd1' images
>>
>
'1d01c6c8-b61e-42bc-a054-f04c3f792b10'..'ef6f732e-2a7a-4a14-a10f-bcc88bdd805f'
>> failed. Images have been marked illegal and can no longer be
>> previewed
>> or reverted to. Please retry Live Merge on the snapshot to complete
>> the operation.
>>
>> On Mon, Jun 18, 2018 at 5:46 PM, <nicolas(a)devels.es> wrote:
>>
>> Indeed, when the problem started I think the SPM was the host I
>> added as VDSM log in the first e-mail. Currently it is the one I
>> sent in the second mail.
>>
>> FWIW, if it helps to debug more fluently, we can provide VPN access
>> to our infrastructure so you can access and see whateve you need
>> (all hosts, DB, etc...).
>>
>> Right now the machines that keep running work, but once shut down
>> they start showing the problem below...
>>
>> Thank you
>>
>> El 2018-06-18 15:20, Benny Zlotnik escribió:
>>
>> I'm having trouble following the errors, I think the SPM changed or
>> the vdsm log from the right host might be missing.
>>
>> However, I believe what started the problems is this transaction
>> timeout:
>>
>> 2018-06-15 14:20:51,378+01 ERROR
>> [org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
>> (org.ovirt.thread.pool-6-thread-29)
>> [1db468cb-85fd-4189-b356-d31781461504] [within thread]: endAction
>> for
>> action type RemoveSnapshotSingleDisk threw an exception.:
>> org.springframework.jdbc.CannotGetJdbcConnectionException: Could
>> not
>> get JDBC Connection; nested exception is java.sql.SQLException:
>> javax.resource.ResourceException: IJ000460: Error checking for a
>> transaction
>> at
>>
>>
>
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
>> [spring-jdbc.jar:4.2.4.RELEASE]
>> at
>>
>>
> org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
>> [spring-jdbc.jar:4.2.4.RELEASE]
>> at
>>
>>
> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680)
>> [spring-jdbc.jar:4.2.4.RELEASE]
>> at
>>
>>
> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712)
>> [spring-jdbc.jar:4.2.4.RELEASE]
>> at
>>
>>
> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:762)
>> [spring-jdbc.jar:4.2.4.RELEASE]
>> at
>>
>>
>
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:152)
>> [dal.jar:]
>>
>> This looks like a bug
>>
>> Regardless, I am not sure restoring a backup would help since you
>> probably have orphaned images on the storage which need to be
>> removed
>>
>> Adding Ala
>>
>> On Mon, Jun 18, 2018 at 4:19 PM, <nicolas(a)devels.es> wrote:
>>
>> Hi Benny,
>>
>> Please find the SPM logs at [1].
>>
>> Thank you
>>
>> [1]:
>>
>>
>
https://wetransfer.com/downloads/62bf649462aabbc2ef21824682b0a08320180618...
>> [2]
>> [1]
>> [1]
>>
>> El 2018-06-18 13:19, Benny Zlotnik escribió:
>> Can you send the SPM logs as well?
>>
>> On Mon, Jun 18, 2018 at 1:13 PM, <nicolas(a)devels.es> wrote:
>>
>> Hi Benny,
>>
>> Please find the logs at [1].
>>
>> Thank you.
>>
>> [1]:
>>
>>
>
https://wetransfer.com/downloads/12208fb4a6a5df3114bbbc10af194c8820180618...
>> [3]
>>
>> [2]
>> [2]
>>
>> [1]
>>
>> El 2018-06-18 09:28, Benny Zlotnik escribió:
>>
>> Can you provide full engine and vdsm logs?
>>
>> On Mon, Jun 18, 2018 at 11:20 AM, <nicolas(a)devels.es> wrote:
>>
>> Hi,
>>
>> We're running oVirt 4.1.9 (we cannot upgrade at this time) and
>> we're having a major problem in our infrastructure. On friday, a
>> snapshots were automatically created on more than 200 VMs and as
>> this was just a test task, all of them were deleted at the same
>> time, which seems to have corrupted several VMs.
>>
>> When trying to delete a snapshot on some of the VMs, a "General
>> error" is thrown with a NullPointerException in the engine log
>> (attached).
>>
>> But the worst part is that when some of these machines is powered
>> off and then powered on, the VMs are corrupt...
>>
>> VM myvm is down with error. Exit message: Bad volume specification
>> {u'index': 0, u'domainID':
u'110ea376-d789-40a1-b9f6-6b40c31afe01',
>> 'reqsize': '0', u'format': u'cow',
u'bootOrder': u'1', u'address':
>> {u'function': u'0x0', u'bus': u'0x00',
u'domain': u'0x0000',
>> u'type': u'pci', u'slot': u'0x06'},
u'volumeID':
>> u'1fd0f9aa-6505-45d2-a17e-859bd5dd4290', 'apparentsize':
>> '23622320128', u'imageID':
u'65519220-68e1-462a-99b3-f0763c78eae2',
>> u'discard': False, u'specParams': {}, u'readonly':
u'false',
>> u'iface': u'virtio', u'optional': u'false',
u'deviceId':
>> u'65519220-68e1-462a-99b3-f0763c78eae2', 'truesize':
'23622320128',
>> u'poolID': u'75bf8f48-970f-42bc-8596-f8ab6efb2b63',
u'device':
>> u'disk', u'shared': u'false', u'propagateErrors':
u'off', u'type':
>> u'disk'}.
>>
>> We're really frustrated by now and don't know how to procceed... We
>> have a DB backup (with engine-backup) from thursday which would
>> have
>> a "sane" DB definition without all the snapshots, as they were all
>> created on friday. Would it be safe to restore this backup?
>>
>> Any help is really appreciated...
>>
>> Thanks.
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ [4]
>> [3]
>> [3]
>> [2]
>> [1]
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/ [5] [4]
>> [4]
>> [3]
>> [2]
>> List Archives:
>>
>>
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P5OOGBL3BRZ...
>> [6]
>> [5]
>> [5]
>> [4]
>> [3]
>>
>> Links:
>> ------
>> [1] https://www.ovirt.org/site/privacy-policy/ [4] [3] [3] [2]
>> [2] https://www.ovirt.org/community/about/community-guidelines/ [5]
>> [4]
>> [4]
>> [3]
>> [3]
>>
>>
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P5OOGBL3BRZ...
>> [6]
>> [5]
>> [5]
>> [4]
>>
>> Links:
>> ------
>> [1]
>>
>>
>
https://wetransfer.com/downloads/12208fb4a6a5df3114bbbc10af194c8820180618...
>> [3]
>> [2]
>> [2]
>> [2] https://www.ovirt.org/site/privacy-policy/ [4] [3] [3]
>> [3] https://www.ovirt.org/community/about/community-guidelines/ [5]
>> [4]
>> [4]
>> [4]
>>
>>
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P5OOGBL3BRZ...
>> [6]
>> [5]
>> [5]
>>
>> Links:
>> ------
>> [1]
>>
>>
>
https://wetransfer.com/downloads/62bf649462aabbc2ef21824682b0a08320180618...
>> [2]
>> [1]
>> [2]
>>
>>
>
https://wetransfer.com/downloads/12208fb4a6a5df3114bbbc10af194c8820180618...
>> [3]
>> [2]
>> [3] https://www.ovirt.org/site/privacy-policy/ [4] [3]
>> [4] https://www.ovirt.org/community/about/community-guidelines/ [5]
>> [4]
>> [5]
>>
>>
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P5OOGBL3BRZ...
>> [6]
>> [5]
>>
>> Links:
>> ------
>> [1]
>>
>
https://wetransfer.com/downloads/62bf649462aabbc2ef21824682b0a08320180618...
>> [2]
>> [2]
>>
>
https://wetransfer.com/downloads/12208fb4a6a5df3114bbbc10af194c8820180618...
>> [3]
>> [3] https://www.ovirt.org/site/privacy-policy/ [4]
>> [4] https://www.ovirt.org/community/about/community-guidelines/ [5]
>> [5]
>>
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P5OOGBL3BRZ...
>> [6]
>
>
>
> Links:
> ------
> [1] http://host.domain.es
> [2]
>
https://wetransfer.com/downloads/62bf649462aabbc2ef21824682b0a08320180618...
> [3]
>
https://wetransfer.com/downloads/12208fb4a6a5df3114bbbc10af194c8820180618...
> [4] https://www.ovirt.org/site/privacy-policy/
> [5] https://www.ovirt.org/community/about/community-guidelines/
> [6]
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P5OOGBL3BRZ...