[ovirt-users] missing disk after storage domain expansion
Maor Lipchuk
mlipchuk at redhat.com
Tue May 19 16:23:13 UTC 2015
Hi Andrea,
I'm using "postman" chrome-extension to execute my REST requests.
For firefox the rest-client add-on you can use is
https://addons.mozilla.org/de/firefox/addon/restclient/
Juan, do we have a wiki for step by step instructions of REST
Regards,
Maor
----- Original Message -----
> From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
> To: "Maor Lipchuk" <mlipchuk at redhat.com>
> Cc: users at ovirt.org
> Sent: Tuesday, May 19, 2015 4:46:09 PM
> Subject: RE: [ovirt-users] missing disk after storage domain expansion
>
> I got an error but I'm not 100% sure I did everything correctly
> Error below:
> <html><head><title>JBoss Web/7.0.13.Final - Error
> report</title><style><!--H1
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
> H2
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
> H3
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
> BODY
> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
> P
> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
> {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
> </head><body><h1>HTTP Status 415 - Cannot consume content type</h1><HR
> size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b>
> <u>Cannot consume content type</u></p><p><b>description</b> <u>The server
> refused this request because the request entity is in a format not supported
> by the requested resource for the requested method (Cannot consume content
> type).</u></p><HR size="1" noshade="noshade"><h3>JBoss
> Web/7.0.13.Final</h3></body></html>
>
> Actions:
> Used RESTclient 2.3.0 for Firefox to POST this body
> <disk id='16736ce0-a9df-410f-9f29-3a28364cdd41'></disk>
>
> I think I need a step by step guide about how to use the REST client to POST
> commands to ovirt service remotely
>
> Cheers
> AG
>
> -----Original Message-----
> From: Maor Lipchuk [mailto:mlipchuk at redhat.com]
> Sent: Monday, May 18, 2015 4:50 PM
> To: Andrea Ghelardi
> Subject: Re: [ovirt-users] missing disk after storage domain expansion
>
> I've went through the logs, though I didn't noticed any special information.
> Please let me know how the registration of the disk went.
>
> Regards,
> Maor
>
>
>
> ----- Original Message -----
> > From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
> > To: "Maor Lipchuk" <mlipchuk at redhat.com>
> > Sent: Monday, May 18, 2015 4:35:30 PM
> > Subject: RE: [ovirt-users] missing disk after storage domain expansion
> >
> > Here the full set of logs as promised.
> >
> > Unfortunately, I do not think I have vdsm logs pointing to that date
> > (logs rotated already).
> > I tried to search for
> > xzgrep 16736ce0-a9df-410f-9f29-3a28364cdd41 /var/log/vdsm/* but
> > nothing found
> >
> > cheers
> > AG
> >
> > -----Original Message-----
> > From: Andrea Ghelardi [mailto:a.ghelardi at iontrading.com]
> > Sent: Monday, May 18, 2015 3:30 PM
> > To: 'Maor Lipchuk'
> > Cc: 'users at ovirt.org'
> > Subject: RE: [ovirt-users] missing disk after storage domain expansion
> >
> > Hi Maor,
> >
> > 1) disk creation: ok
> > 2) no free space at the moment of creation: this is very strange! Are
> > you sure? I assure you the disk was working, exported to VM which
> > formatted and wrote data on it
> > 3) no disk removal mention: this is the very issue I'm facing. In fact
> > disk disappeared with no notification, no logs, nothing. I'm
> > forwarding you full set of engine.log separately. If you need
> > vdsm.logs just tell me from which server you need it as I have several
> > cluster.
> > 4) I have not yet tried to register using the procedure you provided
> > me earlier. I'll try tomorrow.
> >
> > Thanks
> > AG
> >
> >
> > -----Original Message-----
> > From: Maor Lipchuk [mailto:mlipchuk at redhat.com]
> > Sent: Monday, May 18, 2015 1:15 PM
> > To: Andrea Ghelardi
> > Cc: users at ovirt.org
> > Subject: Re: [ovirt-users] missing disk after storage domain expansion
> >
> > Hi Andrea,
> >
> > I guessed I missed those logs, I found it now, thanks.
> >
> > I do see the disk creation in the log from 20150501, I can also see
> > that the Storage Domain hertz-dstore2 has 0 GB of free space at the
> > moment of
> > creation:
> > 2015-05-01 03:30:05,801 ERROR
> > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> > (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack:
> > null, Custom Event ID: -1, Message: Critical, Low disk space.
> > hertz-dstore2 domain has 0 GB of free space
> >
> > though, I don't see any indication of the removed disks from the engine.
> > I have also noticed a gap from the first log which finished at
> > 2015-05-01
> > 03:30:05 and the other log which starts at 2015-05-04 03:22:19,879.
> >
> > Did you try to register the disk as described in the wiki?
> >
> > Regards,
> > Maor
> >
> >
> >
> >
> > ----- Original Message -----
> > > From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
> > > To: "Maor Lipchuk" <mlipchuk at redhat.com>
> > > Cc: users at ovirt.org
> > > Sent: Monday, May 18, 2015 12:10:55 PM
> > > Subject: RE: [ovirt-users] missing disk after storage domain
> > > expansion
> > >
> > > Hi Maor,
> > > I already added logs in my previous email, did you received them?
> > > I'm sending them again, but only privately to you not to burden the
> > > mailing list (or let me know if you prefer otherwise).
> > >
> > > Cheers
> > > AG
> > >
> > >
> > > -----Original Message-----
> > > From: Maor Lipchuk [mailto:mlipchuk at redhat.com]
> > > Sent: Saturday, May 16, 2015 10:03 AM
> > > To: Andrea Ghelardi
> > > Cc: users at ovirt.org
> > > Subject: Re: [ovirt-users] missing disk after storage domain
> > > expansion
> > >
> > > Hi Andrea,
> > >
> > > The OVF_STORE disks are disks which mainly being used internally in
> > > the engine to store VMs' and Templates' OVFs, you can disregard them
> > > for now.
> > >
> > > The behavior you mentioned, that suddenly the disk has disappeared
> > > from the setup, sounds very weird and I would like to investigate
> > > this a bit more.
> > > if you can please add the engine and vdsm logs, I can know what made
> > > the disk get removed from the setup, at the first place.
> > > Could it be that some one accessed the data base by any chance?
> > >
> > > Regarding the disk registration, basically it should not create any
> > > problems unless, if your disk was deleted manually form the DB, and
> > > there are still leftovers related to the disk in some of the tables.
> > > Once the disk is registered to the setup, you can try to attach it
> > > to the VM, and try to run it.
> > >
> > >
> > > Regards,
> > > Maor
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
> > > > To: "Maor Lipchuk" <mlipchuk at redhat.com>
> > > > Cc: users at ovirt.org
> > > > Sent: Friday, May 15, 2015 4:54:25 PM
> > > > Subject: RE: [ovirt-users] missing disk after storage domain
> > > > expansion
> > > >
> > > > After querying the hosted engine with the suggested command
> > > > https://pisa-ion-ovirtm-01/ovirt-engine/api/storagedomains/7a48fe4
> > > > 6-
> > > > 21 12-40a4-814f-24d74c760b2d/disks;unregistered
> > > >
> > > > Indeed shows a (unregistered?) disk
> > > >
> > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <disks>
> > > > <disk
> > > > href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41"
> > > > id="16736ce0-a9df-410f-9f29-3a28364cdd41">
> > > > <actions>
> > > > <link
> > > > href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/export"
> > > > rel="export"/>
> > > > </actions>
> > > > <name>hertz_disk5</name>
> > > > <description>disk for SYBASE installation, on SAN shared
> > > > storage</description> <link
> > > > href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/permissions"
> > > > rel="permissions"/>
> > > > <link
> > > > href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/statistics"
> > > > rel="statistics"/>
> > > > <alias>hertz_disk5</alias>
> > > > <image_id>4ab070c0-fb16-452d-8521-4ff0b004aef3</image_id>
> > > > <storage_domain
> > > > href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d"
> > > > id="7a48fe46-2112-40a4-814f-24d74c760b2d"/>
> > > > <storage_domains>
> > > > <storage_domain id="7a48fe46-2112-40a4-814f-24d74c760b2d"/>
> > > > </storage_domains>
> > > > <size>225485783040</size>
> > > > <provisioned_size>225485783040</provisioned_size>
> > > > <actual_size>225485783040</actual_size>
> > > > <status>
> > > > <state>ok</state>
> > > > </status>
> > > > <interface>ide</interface>
> > > > <format>raw</format>
> > > > <sparse>false</sparse>
> > > > <bootable>false</bootable>
> > > > <shareable>false</shareable>
> > > > <wipe_after_delete>false</wipe_after_delete>
> > > > <propagate_errors>false</propagate_errors>
> > > > </disk>
> > > > </disks>
> > > >
> > > > I'm waiting for any comments on the two OVF disk or any advice
> > > > before proceeding with the registering command (test in production
> > > > environment hmmmmm)
> > > >
> > > > Cheers
> > > > AG
> > > >
> > > > -----Original Message-----
> > > > From: Andrea Ghelardi [mailto:a.ghelardi at iontrading.com]
> > > > Sent: Friday, May 15, 2015 12:30 PM
> > > > To: Maor Lipchuk
> > > > Cc: users at ovirt.org
> > > > Subject: RE: [ovirt-users] missing disk after storage domain
> > > > expansion
> > > >
> > > > Thank you for your reply!
> > > > I'm unsure if the disk contained any snapshots. I do not think so.
> > > > Is the register action a safe one to do in production system? I
> > > > wouldn't mess any of my existing running servers.
> > > >
> > > > Here the relevant logs from the date
> > > > ./engine.log-20150501.gz:2015-04-29 16:10:40,868 : disk creation
> > > > ("hertz_disk5" newImageId = 4ab070c0-fb16-452d-8521-4ff0b004aef3)
> > > > ./engine.log-20150512.gz:2015-05-04 17:22:42,691 : last
> > > > occurrence of the imageid
> > > >
> > > > Please note from
> > > > ./engine.log-20150512.gz 2015-05-11 12:07:16,719 : creation of
> > > > two disk OVF store (??) on the same storage domain
> > > > 7a48fe46-2112-40a4-814f-24d74c760b2d
> > > > right after the volume expansion
> > > >
> > > > In the instructions.txt what I did to enlarge the volume (and
> > > > lose the
> > > > disk) (the guides is informational only, I followed the steps on a
> > > > different
> > > > server)
> > > >
> > > > Here some cats to thank you in advance
> > > > https://www.youtube.com/watch?v=nfTY41-zC7Y
> > >
> > >
> > > lol :)
> > > Nice cats
> > >
> > > > AG
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Maor Lipchuk [mailto:mlipchuk at redhat.com]
> > > > Sent: Wednesday, May 13, 2015 11:02 PM
> > > > To: Andrea Ghelardi
> > > > Cc: users at ovirt.org
> > > > Subject: Re: [ovirt-users] missing disk after storage domain
> > > > expansion
> > > >
> > > > Hi Andrea,
> > > >
> > > > First of all, the issue sounds quite severe, can u please attach
> > > > the engine logs so we can try to figure out how that happened.
> > > > second, does this disk contained any snapshots?
> > > > If not, can you try to register it back (see
> > > > http://www.ovirt.org/Features/ImportStorageDomain#Register_an_unre
> > > > gi
> > > > st
> > > > ered_disk)
> > > >
> > > >
> > > > Regards,
> > > > Maor
> > > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > From: "Andrea Ghelardi" <a.ghelardi at iontrading.com>
> > > > > To: users at ovirt.org
> > > > > Sent: Monday, May 11, 2015 12:01:17 AM
> > > > > Subject: Re: [ovirt-users] missing disk after storage domain
> > > > > expansion
> > > > >
> > > > >
> > > > >
> > > > > Ok, so,
> > > > >
> > > > > After _ a lot _ of unsuccessful approach, I finally connected to
> > > > > postegre DB directly.
> > > > >
> > > > > Browsing the tables I found “unregistered_ovf_of_entities” where
> > > > > there is a reference of the missing disk
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > @ovf:diskId:4ab070c0-fb16-452d-8521-4ff0b004aef3
> > > > >
> > > > > @ovf:size:210
> > > > >
> > > > > @ovf:actual_size:210
> > > > >
> > > > > @ovf:vm_snapshot_id:2e24b255-bb84-4284-8785-e2a042045882
> > > > >
> > > > > @ovf:fileRef:16736ce0-a9df-410f-9f29-3a28364cdd41/4ab070c0-fb16-
> > > > > 45
> > > > > 2d
> > > > > -8
> > > > > 521-4ff0b004aef3
> > > > >
> > > > > @ovf:format:
> > > > > http://www.vmware.com/specifications/vmdk.html#sparse
> > > > >
> > > > > @ovf:volume-format:RAW
> > > > >
> > > > > @ovf:volume-type:Preallocated
> > > > >
> > > > > @ovf:disk-interface:VirtIO
> > > > >
> > > > > @ovf:boot:false
> > > > >
> > > > > @ovf:disk-alias:hertz_disk5
> > > > >
> > > > > @ovf:disk-description:disk for SYBASE installation, on SAN
> > > > > shared storage
> > > > >
> > > > > @ovf:wipe-after-delete:false
> > > > >
> > > > >
> > > > >
> > > > > However, I’ve been unable to find any other helpful details.
> > > > >
> > > > > I guess the disk is not recoverable at this point?
> > > > >
> > > > > Any guru who has a good ovirt DB kwnoledge willing to give me
> > > > > some advice?
> > > > >
> > > > >
> > > > >
> > > > > Thanks as usual
> > > > >
> > > > > AG
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Andrea Ghelardi [mailto: a.ghelardi at iontrading.com ]
> > > > > Sent: Thursday, May 07, 2015 6:08 PM
> > > > > To: ' users at ovirt.org '
> > > > > Subject: missing disk after storage domain expansion
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Hi gentlemen,
> > > > >
> > > > >
> > > > >
> > > > > I recently found an error on 1 of my storages: it was
> > > > > complaining about no free space but the VM was running and disk
> > > > > operational.
> > > > >
> > > > > Since I needed to perform some maintenance on the VM, I shut it
> > > > > down and at restart VM couldn’t boot up properly.
> > > > >
> > > > > Checked VM via console and a disk was missing. Edited fstab
> > > > > (luckily this disk was not root but heck! It had a Sybase DB on
> > > > > it!) and restarted VM this time ok.
> > > > >
> > > > >
> > > > >
> > > > > Since the disk resides on the dstore with no space, I expanded
> > > > > the iSCSI LUN, then refreshed multipath on hosts, then resized
> > > > > PVs and now ovirt is showing the correct size (logs do not
> > > > > complain anymore on no free space).
> > > > >
> > > > >
> > > > >
> > > > > BUUUT
> > > > >
> > > > >
> > > > >
> > > > > Now disk is missing. It is not shown anymore on Disks tab nor
> > > > > anywhere else.
> > > > >
> > > > > Problem is that storage shows 214GB occupancy (size of the
> > > > > missing
> > > > > disk) so data is there but cannot find it anymore.
> > > > >
> > > > >
> > > > >
> > > > > Logs show original disk creation, errors from the lack of space,
> > > > > refresh of the storage size and then.... no more references on
> > > > > the disk.
> > > > >
> > > > >
> > > > >
> > > > > What can I do to find those missing ~210GBs?
> > > > >
> > > > >
> > > > >
> > > > > Cheers
> > > > >
> > > > > Andrea Ghelardi
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Users mailing list
> > > > > Users at ovirt.org
> > > > > http://lists.ovirt.org/mailman/listinfo/users
> > > > >
> > > >
> > >
> >
>
More information about the Users
mailing list