[ovirt-users] missing disk after storage domain expansion [SOLVED]

Wed May 20 10:22:54 UTC 2015

Thanks for all your information.
I created a linux script as per details below.
Attached to this email (some info purged)

Gentlemen, we have a disk now!

I'm marking the post as solved ... even though we do not know where the disk
gone in these weeks yet.
Probably unexpected vacation @Cuba? This disk won't reclaim expenses though

I will create a test VM and perform some tests in the future with the
expansion feature
Thank you for all your support
More cats: https://www.youtube.com/watch?v=cWKmorjUzF0
AG

-----Original Message-----
From: Juan Hernández [mailto:jhernand at redhat.com]
Sent: Wednesday, May 20, 2015 10:37 AM
To: Maor Lipchuk; Andrea Ghelardi
Cc: users at ovirt.org
Subject: Re: [ovirt-users] missing disk after storage domain expansion

On 05/19/2015 06:23 PM, Maor Lipchuk wrote:
> Hi Andrea,
>
> I'm using "postman" chrome-extension to execute my REST requests.
>
> For firefox the rest-client add-on you can use is
> https://addons.mozilla.org/de/firefox/addon/restclient/
>
> Juan, do we have a wiki for step by step instructions of REST
>
> Regards,
> Maor
>

I'd suggest to try a command line client, as it is easier to share the
results and check what is going wrong. You can use the "curl" command, for
example. Assuming that you already know the identifiers of the storage
domain and of the disk to register it should be something like this:

---8<---
#!/bin/sh -ex

url="https://yourengine/ovirt-engine/api"
user="your user"
password="your password"
storage_domain_id="the storage domain id"
disk_id="the disk id"

curl \
--verbose \
--insecure \
--user "${user}:${password}" \
--request POST \
--header "Content-Type: application/xml" \ --header "Accept:
application/xml" \ --data "
<disk id=\"${disk_id}\"/>
" \
"${url}/storagedomains/${storage_domain_id}/disks;unregistered"
--->8---

>
> ----- Original Message -----
>> From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
>> To: "Maor Lipchuk" <mlipchuk at redhat.com>
>> Cc: users at ovirt.org
>> Sent: Tuesday, May 19, 2015 4:46:09 PM
>> Subject: RE: [ovirt-users] missing disk after storage domain
>> expansion
>>
>> I got an error but I'm not 100% sure I did everything correctly Error
>> below:
>> <html><head><title>JBoss Web/7.0.13.Final - Error
>> report</title><style><!--H1
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52
>> 5D76;font-size:22px;}
>> H2
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52
>> 5D76;font-size:16px;}
>> H3
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52
>> 5D76;font-size:14px;}
>> BODY
>> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:whi
>> te;} B
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52
>> 5D76;}
>> P
>> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;fon
>> t-size:12px;}A {color : black;}A.name {color : black;}HR {color :
>> #525D76;}--></style> </head><body><h1>HTTP Status 415 - Cannot
>> consume content type</h1><HR size="1"
>> noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b>
>> <u>Cannot consume content type</u></p><p><b>description</b> <u>The
>> server refused this request because the request entity is in a format
>> not supported by the requested resource for the requested method
>> (Cannot consume content type).</u></p><HR size="1"
>> noshade="noshade"><h3>JBoss Web/7.0.13.Final</h3></body></html>
>>
>> Actions:
>> Used RESTclient 2.3.0 for Firefox to POST this body <disk
>> id='16736ce0-a9df-410f-9f29-3a28364cdd41'></disk>
>>
>> I think I need a step by step guide about how to use the REST client
>> to POST commands to ovirt service remotely
>>
>> Cheers
>> AG
>>
>> -----Original Message-----
>> From: Maor Lipchuk [mailto:mlipchuk at redhat.com]
>> Sent: Monday, May 18, 2015 4:50 PM
>> To: Andrea Ghelardi
>> Subject: Re: [ovirt-users] missing disk after storage domain
>> expansion
>>
>> I've went through the logs, though I didn't noticed any special
>> information.
>> Please let me know how the registration of the disk went.
>>
>> Regards,
>> Maor
>>
>>
>>
>> ----- Original Message -----
>>> From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
>>> To: "Maor Lipchuk" <mlipchuk at redhat.com>
>>> Sent: Monday, May 18, 2015 4:35:30 PM
>>> Subject: RE: [ovirt-users] missing disk after storage domain
>>> expansion
>>>
>>> Here the full set of logs as promised.
>>>
>>> Unfortunately, I do not think I have vdsm logs pointing to that date
>>> (logs rotated already).
>>> I tried to search for
>>> xzgrep 16736ce0-a9df-410f-9f29-3a28364cdd41 /var/log/vdsm/* but
>>> nothing found
>>>
>>> cheers
>>> AG
>>>
>>> -----Original Message-----
>>> From: Andrea Ghelardi [mailto:a.ghelardi at iontrading.com]
>>> Sent: Monday, May 18, 2015 3:30 PM
>>> To: 'Maor Lipchuk'
>>> Cc: 'users at ovirt.org'
>>> Subject: RE: [ovirt-users] missing disk after storage domain
>>> expansion
>>>
>>> Hi Maor,
>>>
>>> 1) disk creation: ok
>>> 2) no free space at the moment of creation: this is very strange!
>>> Are you sure? I assure you the disk was working, exported to VM
>>> which formatted and wrote data on it
>>> 3) no disk removal mention: this is the very issue I'm facing. In
>>> fact disk disappeared with no notification, no logs, nothing. I'm
>>> forwarding you full set of engine.log separately. If you need
>>> vdsm.logs just tell me from which server you need it as I have
>>> several cluster.
>>> 4) I have not yet tried to register using the procedure you provided
>>> me earlier. I'll try tomorrow.
>>>
>>> Thanks
>>> AG
>>>
>>>
>>> -----Original Message-----
>>> From: Maor Lipchuk [mailto:mlipchuk at redhat.com]
>>> Sent: Monday, May 18, 2015 1:15 PM
>>> To: Andrea Ghelardi
>>> Cc: users at ovirt.org
>>> Subject: Re: [ovirt-users] missing disk after storage domain
>>> expansion
>>>
>>> Hi Andrea,
>>>
>>> I guessed I missed those logs, I found it now, thanks.
>>>
>>> I do see the disk creation in the log from 20150501, I can also see
>>> that the Storage Domain hertz-dstore2 has 0 GB of free space at the
>>> moment of
>>> creation:
>>> 2015-05-01 03:30:05,801 ERROR
>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirecto
>>> r]
>>> (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack:
>>> null, Custom Event ID: -1, Message: Critical, Low disk space.
>>> hertz-dstore2 domain has 0 GB of free space
>>>
>>> though, I don't see any indication of the removed disks from the engine.
>>> I have also noticed a gap from the first log which finished at
>>> 2015-05-01
>>> 03:30:05 and the other log which starts at 2015-05-04 03:22:19,879.
>>>
>>> Did you try to register the disk as described in the wiki?
>>>
>>> Regards,
>>> Maor
>>>
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
>>>> To: "Maor Lipchuk" <mlipchuk at redhat.com>
>>>> Cc: users at ovirt.org
>>>> Sent: Monday, May 18, 2015 12:10:55 PM
>>>> Subject: RE: [ovirt-users] missing disk after storage domain
>>>> expansion
>>>>
>>>> Hi Maor,
>>>> I already added logs in my previous email, did you received them?
>>>> I'm sending them again, but only privately to you not to burden the
>>>> mailing list (or let me know if you prefer otherwise).
>>>>
>>>> Cheers
>>>> AG
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Maor Lipchuk [mailto:mlipchuk at redhat.com]
>>>> Sent: Saturday, May 16, 2015 10:03 AM
>>>> To: Andrea Ghelardi
>>>> Cc: users at ovirt.org
>>>> Subject: Re: [ovirt-users] missing disk after storage domain
>>>> expansion
>>>>
>>>> Hi Andrea,
>>>>
>>>> The OVF_STORE disks are disks which mainly being used internally in
>>>> the engine to store VMs' and Templates' OVFs, you can disregard
>>>> them for now.
>>>>
>>>> The behavior you mentioned, that suddenly the disk has disappeared
>>>> from the setup, sounds very weird and I would like to investigate
>>>> this a bit more.
>>>> if you can please add the engine and vdsm logs, I can know what
>>>> made the disk get removed from the setup, at the first place.
>>>> Could it be that some one accessed the data base by any chance?
>>>>
>>>> Regarding the disk registration, basically it should not create any
>>>> problems unless, if your disk was deleted manually form the DB, and
>>>> there are still leftovers related to the disk in some of the tables.
>>>> Once the disk is registered to the setup, you can try to attach it
>>>> to the VM, and try to run it.
>>>>
>>>>
>>>> Regards,
>>>> Maor
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
>>>>> To: "Maor Lipchuk" <mlipchuk at redhat.com>
>>>>> Cc: users at ovirt.org
>>>>> Sent: Friday, May 15, 2015 4:54:25 PM
>>>>> Subject: RE: [ovirt-users] missing disk after storage domain
>>>>> expansion
>>>>>
>>>>> After querying the hosted engine with the suggested command
>>>>> https://pisa-ion-ovirtm-01/ovirt-engine/api/storagedomains/7a48fe4
>>>>> 6-
>>>>> 21 12-40a4-814f-24d74c760b2d/disks;unregistered
>>>>>
>>>>> Indeed shows  a (unregistered?) disk
>>>>>
>>>>> <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <disks>
>>>>> <disk
>>>>> href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41"
>>>>> id="16736ce0-a9df-410f-9f29-3a28364cdd41">
>>>>> <actions>
>>>>> <link
>>>>> href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/export"
>>>>> rel="export"/>
>>>>> </actions>
>>>>> <name>hertz_disk5</name>
>>>>> <description>disk for SYBASE installation, on SAN shared
>>>>> storage</description> <link
>>>>> href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/permissions"
>>>>> rel="permissions"/>
>>>>> <link
>>>>> href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/statistics"
>>>>> rel="statistics"/>
>>>>> <alias>hertz_disk5</alias>
>>>>> <image_id>4ab070c0-fb16-452d-8521-4ff0b004aef3</image_id>
>>>>> <storage_domain
>>>>> href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d"
>>>>> id="7a48fe46-2112-40a4-814f-24d74c760b2d"/>
>>>>> <storage_domains>
>>>>> <storage_domain id="7a48fe46-2112-40a4-814f-24d74c760b2d"/>
>>>>> </storage_domains>
>>>>> <size>225485783040</size>
>>>>> <provisioned_size>225485783040</provisioned_size>
>>>>> <actual_size>225485783040</actual_size>
>>>>> <status>
>>>>> <state>ok</state>
>>>>> </status>
>>>>> <interface>ide</interface>
>>>>> <format>raw</format>
>>>>> <sparse>false</sparse>
>>>>> <bootable>false</bootable>
>>>>> <shareable>false</shareable>
>>>>> <wipe_after_delete>false</wipe_after_delete>
>>>>> <propagate_errors>false</propagate_errors>
>>>>> </disk>
>>>>> </disks>
>>>>>
>>>>> I'm waiting for any comments on the two OVF disk or any advice
>>>>> before proceeding with the registering command (test in production
>>>>> environment hmmmmm)
>>>>>
>>>>> Cheers
>>>>> AG
>>>>>
>>>>> -----Original Message-----
>>>>> From: Andrea Ghelardi [mailto:a.ghelardi at iontrading.com]
>>>>> Sent: Friday, May 15, 2015 12:30 PM
>>>>> To: Maor Lipchuk
>>>>> Cc: users at ovirt.org
>>>>> Subject: RE: [ovirt-users] missing disk after storage domain
>>>>> expansion
>>>>>
>>>>> Thank you for your reply!
>>>>> I'm unsure if the disk contained any snapshots. I do not think so.
>>>>> Is the register action a safe one to do in production system? I
>>>>> wouldn't mess any of my existing running servers.
>>>>>
>>>>> Here the relevant logs from the date
>>>>> ./engine.log-20150501.gz:2015-04-29 16:10:40,868  : disk creation
>>>>> ("hertz_disk5" newImageId = 4ab070c0-fb16-452d-8521-4ff0b004aef3)
>>>>> ./engine.log-20150512.gz:2015-05-04 17:22:42,691  : last
>>>>> occurrence of the imageid
>>>>>
>>>>> Please note from
>>>>> ./engine.log-20150512.gz  2015-05-11 12:07:16,719 : creation of
>>>>> two disk OVF store (??) on the same storage domain
>>>>> 7a48fe46-2112-40a4-814f-24d74c760b2d
>>>>> right after the volume expansion
>>>>>
>>>>> In the instructions.txt  what I did to enlarge the volume (and
>>>>> lose the
>>>>> disk) (the guides is informational only, I followed the steps on a
>>>>> different
>>>>> server)
>>>>>
>>>>> Here some cats to thank you in advance
>>>>> https://www.youtube.com/watch?v=nfTY41-zC7Y
>>>>
>>>>
>>>> lol :)
>>>> Nice cats
>>>>
>>>>> AG
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Maor Lipchuk [mailto:mlipchuk at redhat.com]
>>>>> Sent: Wednesday, May 13, 2015 11:02 PM
>>>>> To: Andrea Ghelardi
>>>>> Cc: users at ovirt.org
>>>>> Subject: Re: [ovirt-users] missing disk after storage domain
>>>>> expansion
>>>>>
>>>>> Hi Andrea,
>>>>>
>>>>> First of all, the issue sounds quite severe, can u please attach
>>>>> the engine logs so we can try to figure out how that happened.
>>>>> second, does this disk contained any snapshots?
>>>>>  If not, can you try to register it back (see
>>>>> http://www.ovirt.org/Features/ImportStorageDomain#Register_an_unre
>>>>> gi
>>>>> st
>>>>> ered_disk)
>>>>>
>>>>>
>>>>> Regards,
>>>>> Maor
>>>>>
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
>>>>>> To: users at ovirt.org
>>>>>> Sent: Monday, May 11, 2015 12:01:17 AM
>>>>>> Subject: Re: [ovirt-users] missing disk after storage domain
>>>>>> expansion
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ok, so,
>>>>>>
>>>>>> After _ a lot _ of unsuccessful approach, I finally connected to
>>>>>> postegre DB directly.
>>>>>>
>>>>>> Browsing the tables I found “unregistered_ovf_of_entities” where
>>>>>> there is a reference of the missing disk
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> @ovf:diskId:4ab070c0-fb16-452d-8521-4ff0b004aef3
>>>>>>
>>>>>> @ovf:size:210
>>>>>>
>>>>>> @ovf:actual_size:210
>>>>>>
>>>>>> @ovf:vm_snapshot_id:2e24b255-bb84-4284-8785-e2a042045882
>>>>>>
>>>>>> @ovf:fileRef:16736ce0-a9df-410f-9f29-3a28364cdd41/4ab070c0-fb16-
>>>>>> 45
>>>>>> 2d
>>>>>> -8
>>>>>> 521-4ff0b004aef3
>>>>>>
>>>>>> @ovf:format:
>>>>>> http://www.vmware.com/specifications/vmdk.html#sparse
>>>>>>
>>>>>> @ovf:volume-format:RAW
>>>>>>
>>>>>> @ovf:volume-type:Preallocated
>>>>>>
>>>>>> @ovf:disk-interface:VirtIO
>>>>>>
>>>>>> @ovf:boot:false
>>>>>>
>>>>>> @ovf:disk-alias:hertz_disk5
>>>>>>
>>>>>> @ovf:disk-description:disk for SYBASE installation, on SAN shared
>>>>>> storage
>>>>>>
>>>>>> @ovf:wipe-after-delete:false
>>>>>>
>>>>>>
>>>>>>
>>>>>> However, I’ve been unable to find any other helpful details.
>>>>>>
>>>>>> I guess the disk is not recoverable at this point?
>>>>>>
>>>>>> Any guru who has a good ovirt DB kwnoledge willing to give me
>>>>>> some advice?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks as usual
>>>>>>
>>>>>> AG
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: Andrea Ghelardi [mailto: a.ghelardi at iontrading.com ]
>>>>>> Sent: Thursday, May 07, 2015 6:08 PM
>>>>>> To: ' users at ovirt.org '
>>>>>> Subject: missing disk after storage domain expansion
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi gentlemen,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I recently found an error on 1 of my storages: it was complaining
>>>>>> about no free space but the VM was running and disk operational.
>>>>>>
>>>>>> Since I needed to perform some maintenance on the VM, I shut it
>>>>>> down and at restart VM couldn’t boot up properly.
>>>>>>
>>>>>> Checked VM via console and a disk was missing. Edited fstab
>>>>>> (luckily this disk was not root but heck! It had a Sybase DB on
>>>>>> it!) and restarted VM this time ok.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Since the disk resides on the dstore with no space, I expanded
>>>>>> the iSCSI LUN, then refreshed multipath on hosts, then resized
>>>>>> PVs and now ovirt is showing the correct size (logs do not
>>>>>> complain anymore on no free space).
>>>>>>
>>>>>>
>>>>>>
>>>>>> BUUUT
>>>>>>
>>>>>>
>>>>>>
>>>>>> Now disk is missing. It is not shown anymore on Disks tab nor
>>>>>> anywhere else.
>>>>>>
>>>>>> Problem is that storage shows 214GB occupancy (size of the
>>>>>> missing
>>>>>> disk) so data is there but cannot find it anymore.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Logs show original disk creation, errors from the lack of space,
>>>>>> refresh of the storage size and then.... no more references on
>>>>>> the disk.
>>>>>>
>>>>>>
>>>>>>
>>>>>> What can I do to find those missing ~210GBs?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Andrea Ghelardi
>>>>>>

--
Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD,
28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F.
B82657941 - Red Hat S.L.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: register_disk.sh
Type: application/octet-stream
Size: 449 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150520/6147522a/attachment-0001.obj>