[ovirt-users] missing disk after storage domain expansion

Mon May 18 07:14:56 EDT 2015

Hi Andrea,

I guessed I missed those logs, I found it now, thanks.

I do see the disk creation in the log from 20150501, I can also see that the Storage Domain hertz-dstore2 has 0 GB of free space at the moment of creation:
2015-05-01 03:30:05,801 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. hertz-dstore2 domain has 0 GB of free space

though, I don't see any indication of the removed disks from the engine.
I have also noticed a gap from the first log which finished at 2015-05-01 03:30:05 and the other log which starts at 2015-05-04 03:22:19,879.

Did you try to register the disk as described in the wiki?

Regards,
Maor

----- Original Message -----
> From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
> To: "Maor Lipchuk" <mlipchuk at redhat.com>
> Cc: users at ovirt.org
> Sent: Monday, May 18, 2015 12:10:55 PM
> Subject: RE: [ovirt-users] missing disk after storage domain expansion
> 
> Hi Maor,
> I already added logs in my previous email, did you received them?
> I'm sending them again, but only privately to you not to burden the mailing
> list (or let me know if you prefer otherwise).
> 
> Cheers
> AG
> 
> 
> -----Original Message-----
> From: Maor Lipchuk [mailto:mlipchuk at redhat.com]
> Sent: Saturday, May 16, 2015 10:03 AM
> To: Andrea Ghelardi
> Cc: users at ovirt.org
> Subject: Re: [ovirt-users] missing disk after storage domain expansion
> 
> Hi Andrea,
> 
> The OVF_STORE disks are disks which mainly being used internally in the
> engine to store VMs' and Templates' OVFs, you can disregard them for now.
> 
> The behavior you mentioned, that suddenly the disk has disappeared from the
> setup, sounds very weird and I would like to investigate this a bit more.
> if you can please add the engine and vdsm logs, I can know what made the
> disk get removed from the setup, at the first place.
> Could it be that some one accessed the data base by any chance?
> 
> Regarding the disk registration, basically it should not create any problems
> unless, if your disk was deleted manually form the DB, and there are still
> leftovers related to the disk in some of the tables.
> Once the disk is registered to the setup, you can try to attach it to the
> VM, and try to run it.
> 
> 
> Regards,
> Maor
> 
> 
> 
> ----- Original Message -----
> > From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
> > To: "Maor Lipchuk" <mlipchuk at redhat.com>
> > Cc: users at ovirt.org
> > Sent: Friday, May 15, 2015 4:54:25 PM
> > Subject: RE: [ovirt-users] missing disk after storage domain expansion
> >
> > After querying the hosted engine with the suggested command
> > https://pisa-ion-ovirtm-01/ovirt-engine/api/storagedomains/7a48fe46-21
> > 12-40a4-814f-24d74c760b2d/disks;unregistered
> >
> > Indeed shows  a (unregistered?) disk
> >
> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <disks> <disk
> > href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41"
> > id="16736ce0-a9df-410f-9f29-3a28364cdd41">
> > <actions>
> > <link
> > href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/export"
> > rel="export"/>
> > </actions>
> > <name>hertz_disk5</name>
> > <description>disk for SYBASE installation, on SAN shared
> > storage</description> <link
> > href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/permissions"
> > rel="permissions"/>
> > <link
> > href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/statistics"
> > rel="statistics"/>
> > <alias>hertz_disk5</alias>
> > <image_id>4ab070c0-fb16-452d-8521-4ff0b004aef3</image_id>
> > <storage_domain
> > href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d"
> > id="7a48fe46-2112-40a4-814f-24d74c760b2d"/>
> > <storage_domains>
> > <storage_domain id="7a48fe46-2112-40a4-814f-24d74c760b2d"/>
> > </storage_domains>
> > <size>225485783040</size>
> > <provisioned_size>225485783040</provisioned_size>
> > <actual_size>225485783040</actual_size>
> > <status>
> > <state>ok</state>
> > </status>
> > <interface>ide</interface>
> > <format>raw</format>
> > <sparse>false</sparse>
> > <bootable>false</bootable>
> > <shareable>false</shareable>
> > <wipe_after_delete>false</wipe_after_delete>
> > <propagate_errors>false</propagate_errors>
> > </disk>
> > </disks>
> >
> > I'm waiting for any comments on the two OVF disk or any advice before
> > proceeding with the registering command (test in production
> > environment hmmmmm)
> >
> > Cheers
> > AG
> >
> > -----Original Message-----
> > From: Andrea Ghelardi [mailto:a.ghelardi at iontrading.com]
> > Sent: Friday, May 15, 2015 12:30 PM
> > To: Maor Lipchuk
> > Cc: users at ovirt.org
> > Subject: RE: [ovirt-users] missing disk after storage domain expansion
> >
> > Thank you for your reply!
> > I'm unsure if the disk contained any snapshots. I do not think so.
> > Is the register action a safe one to do in production system? I
> > wouldn't mess any of my existing running servers.
> >
> > Here the relevant logs from the date
> > ./engine.log-20150501.gz:2015-04-29 16:10:40,868  : disk creation
> > ("hertz_disk5" newImageId = 4ab070c0-fb16-452d-8521-4ff0b004aef3)
> > ./engine.log-20150512.gz:2015-05-04 17:22:42,691  : last  occurrence
> > of the imageid
> >
> > Please note from
> > ./engine.log-20150512.gz  2015-05-11 12:07:16,719 : creation of two
> > disk OVF store (??) on the same storage domain
> > 7a48fe46-2112-40a4-814f-24d74c760b2d
> > right after the volume expansion
> >
> > In the instructions.txt  what I did to enlarge the volume (and lose
> > the
> > disk) (the guides is informational only, I followed the steps on a
> > different
> > server)
> >
> > Here some cats to thank you in advance
> > https://www.youtube.com/watch?v=nfTY41-zC7Y
> 
> 
> lol :)
> Nice cats
> 
> > AG
> >
> >
> > -----Original Message-----
> > From: Maor Lipchuk [mailto:mlipchuk at redhat.com]
> > Sent: Wednesday, May 13, 2015 11:02 PM
> > To: Andrea Ghelardi
> > Cc: users at ovirt.org
> > Subject: Re: [ovirt-users] missing disk after storage domain expansion
> >
> > Hi Andrea,
> >
> > First of all, the issue sounds quite severe, can u please attach the
> > engine logs so we can try to figure out how that happened.
> > second, does this disk contained any snapshots?
> >  If not, can you try to register it back (see
> > http://www.ovirt.org/Features/ImportStorageDomain#Register_an_unregist
> > ered_disk)
> >
> >
> > Regards,
> > Maor
> >
> >
> >
> > ----- Original Message -----
> > > From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
> > > To: users at ovirt.org
> > > Sent: Monday, May 11, 2015 12:01:17 AM
> > > Subject: Re: [ovirt-users] missing disk after storage domain
> > > expansion
> > >
> > >
> > >
> > > Ok, so,
> > >
> > > After _ a lot _ of unsuccessful approach, I finally connected to
> > > postegre DB directly.
> > >
> > > Browsing the tables I found “unregistered_ovf_of_entities” where
> > > there is a reference of the missing disk
> > >
> > >
> > >
> > >
> > >
> > > @ovf:diskId:4ab070c0-fb16-452d-8521-4ff0b004aef3
> > >
> > > @ovf:size:210
> > >
> > > @ovf:actual_size:210
> > >
> > > @ovf:vm_snapshot_id:2e24b255-bb84-4284-8785-e2a042045882
> > >
> > > @ovf:fileRef:16736ce0-a9df-410f-9f29-3a28364cdd41/4ab070c0-fb16-452d
> > > -8
> > > 521-4ff0b004aef3
> > >
> > > @ovf:format: http://www.vmware.com/specifications/vmdk.html#sparse
> > >
> > > @ovf:volume-format:RAW
> > >
> > > @ovf:volume-type:Preallocated
> > >
> > > @ovf:disk-interface:VirtIO
> > >
> > > @ovf:boot:false
> > >
> > > @ovf:disk-alias:hertz_disk5
> > >
> > > @ovf:disk-description:disk for SYBASE installation, on SAN shared
> > > storage
> > >
> > > @ovf:wipe-after-delete:false
> > >
> > >
> > >
> > > However, I’ve been unable to find any other helpful details.
> > >
> > > I guess the disk is not recoverable at this point?
> > >
> > > Any guru who has a good ovirt DB kwnoledge willing to give me some
> > > advice?
> > >
> > >
> > >
> > > Thanks as usual
> > >
> > > AG
> > >
> > >
> > >
> > >
> > > From: Andrea Ghelardi [mailto: a.ghelardi at iontrading.com ]
> > > Sent: Thursday, May 07, 2015 6:08 PM
> > > To: ' users at ovirt.org '
> > > Subject: missing disk after storage domain expansion
> > >
> > >
> > >
> > >
> > > Hi gentlemen,
> > >
> > >
> > >
> > > I recently found an error on 1 of my storages: it was complaining
> > > about no free space but the VM was running and disk operational.
> > >
> > > Since I needed to perform some maintenance on the VM, I shut it down
> > > and at restart VM couldn’t boot up properly.
> > >
> > > Checked VM via console and a disk was missing. Edited fstab (luckily
> > > this disk was not root but heck! It had a Sybase DB on it!) and
> > > restarted VM this time ok.
> > >
> > >
> > >
> > > Since the disk resides on the dstore with no space, I expanded the
> > > iSCSI LUN, then refreshed multipath on hosts, then resized PVs and
> > > now ovirt is showing the correct size (logs do not complain anymore
> > > on no free space).
> > >
> > >
> > >
> > > BUUUT
> > >
> > >
> > >
> > > Now disk is missing. It is not shown anymore on Disks tab nor
> > > anywhere else.
> > >
> > > Problem is that storage shows 214GB occupancy (size of the missing
> > > disk) so data is there but cannot find it anymore.
> > >
> > >
> > >
> > > Logs show original disk creation, errors from the lack of space,
> > > refresh of the storage size and then.... no more references on the disk.
> > >
> > >
> > >
> > > What can I do to find those missing ~210GBs?
> > >
> > >
> > >
> > > Cheers
> > >
> > > Andrea Ghelardi
> > >
> > >
> > >
> > > _______________________________________________
> > > Users mailing list
> > > Users at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/users
> > >
> >
>