New subject: missing disk after storage domain expansion

19 May 2015

      I got an error but I'm not 100% sure I did everything correctly
Error below:
<html><head><title>JBoss Web/7.0.13.Final - Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 415 - Cannot consume content type</h1><HR
size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b>
<u>Cannot consume content type</u></p><p><b>description</b> <u>The server
refused this request because the request entity is in a format not supported
by the requested resource for the requested method (Cannot consume content
type).</u></p><HR size="1" noshade="noshade"><h3>JBoss
Web/7.0.13.Final</h3></body></html>

Actions:
Used RESTclient 2.3.0 for Firefox to POST this body
<disk id='16736ce0-a9df-410f-9f29-3a28364cdd41'></disk>

I think I need a step by step guide about how to use the REST client to POST
commands to ovirt service remotely

Cheers
AG

-----Original Message-----
From: Maor Lipchuk [mailto:mlipchuk@redhat.com]
Sent: Monday, May 18, 2015 4:50 PM
To: Andrea Ghelardi
Subject: Re: [ovirt-users] missing disk after storage domain expansion

I've went through the logs, though I didn't noticed any special information.
Please let me know how the registration of the disk went.

Regards,
Maor

----- Original Message -----
...
From: "Andrea Ghelardi" <a.ghelardi@iontrading.com>
To: "Maor Lipchuk" <mlipchuk@redhat.com>
Sent: Monday, May 18, 2015 4:35:30 PM
Subject: RE: [ovirt-users] missing disk after storage domain expansion
Here the full set of logs as promised.
Unfortunately, I do not think I have vdsm logs pointing to that date
(logs rotated already).
I tried to search for
xzgrep 16736ce0-a9df-410f-9f29-3a28364cdd41 /var/log/vdsm/* but
nothing found
cheers
AG
-----Original Message-----
From: Andrea Ghelardi [mailto:a.ghelardi@iontrading.com]
Sent: Monday, May 18, 2015 3:30 PM
To: 'Maor Lipchuk'
Cc: 'users@ovirt.org'
Subject: RE: [ovirt-users] missing disk after storage domain expansion
Hi Maor,
1) disk creation: ok
2) no free space at the moment of creation: this is very strange! Are
you sure? I assure you the disk was working, exported to VM which
formatted and wrote data on it
3) no disk removal mention: this is the very issue I'm facing. In fact
disk disappeared with no notification, no logs, nothing. I'm
forwarding you full set of engine.log separately. If you need
vdsm.logs just tell me from which server you need it as I have several
cluster.
4) I have not yet tried to register using the procedure you provided
me earlier. I'll try tomorrow.
Thanks
AG
-----Original Message-----
From: Maor Lipchuk [mailto:mlipchuk@redhat.com]
Sent: Monday, May 18, 2015 1:15 PM
To: Andrea Ghelardi
Cc: users@ovirt.org
Subject: Re: [ovirt-users] missing disk after storage domain expansion
Hi Andrea,
I guessed I missed those logs, I found it now, thanks.
I do see the disk creation in the log from 20150501, I can also see
that the Storage Domain hertz-dstore2 has 0 GB of free space at the
moment of
creation:
2015-05-01 03:30:05,801 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack:
null, Custom Event ID: -1, Message: Critical, Low disk space.
hertz-dstore2 domain has 0 GB of free space
though, I don't see any indication of the removed disks from the engine.
I have also noticed a gap from the first log which finished at
2015-05-01
03:30:05 and the other log which starts at 2015-05-04 03:22:19,879.
Did you try to register the disk as described in the wiki?
Regards,
Maor
----- Original Message -----
...
From: "Andrea Ghelardi" <a.ghelardi@iontrading.com>
To: "Maor Lipchuk" <mlipchuk@redhat.com>
Cc: users@ovirt.org
Sent: Monday, May 18, 2015 12:10:55 PM
Subject: RE: [ovirt-users] missing disk after storage domain
expansion
Hi Maor,
I already added logs in my previous email, did you received them?
I'm sending them again, but only privately to you not to burden the
mailing list (or let me know if you prefer otherwise).
Cheers
AG
-----Original Message-----
From: Maor Lipchuk [mailto:mlipchuk@redhat.com]
Sent: Saturday, May 16, 2015 10:03 AM
To: Andrea Ghelardi
Cc: users@ovirt.org
Subject: Re: [ovirt-users] missing disk after storage domain
expansion
Hi Andrea,
The OVF_STORE disks are disks which mainly being used internally in
the engine to store VMs' and Templates' OVFs, you can disregard them
for now.
The behavior you mentioned, that suddenly the disk has disappeared
from the setup, sounds very weird and I would like to investigate
this a bit more.
if you can please add the engine and vdsm logs, I can know what made
the disk get removed from the setup, at the first place.
Could it be that some one accessed the data base by any chance?
Regarding the disk registration, basically it should not create any
problems unless, if your disk was deleted manually form the DB, and
there are still leftovers related to the disk in some of the tables.
Once the disk is registered to the setup, you can try to attach it
to the VM, and try to run it.
Regards,
Maor
----- Original Message -----
...
From: "Andrea Ghelardi" <a.ghelardi@iontrading.com>
To: "Maor Lipchuk" <mlipchuk@redhat.com>
Cc: users@ovirt.org
Sent: Friday, May 15, 2015 4:54:25 PM
Subject: RE: [ovirt-users] missing disk after storage domain
expansion
After querying the hosted engine with the suggested command
https://pisa-ion-ovirtm-01/ovirt-engine/api/storagedomains/7a48fe4
6-
21 12-40a4-814f-24d74c760b2d/disks;unregistered
Indeed shows  a (unregistered?) disk
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <disks>
<disk
href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41"
id="16736ce0-a9df-410f-9f29-3a28364cdd41">
<actions>
<link
href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/export"
rel="export"/>
</actions>
<name>hertz_disk5</name>
<description>disk for SYBASE installation, on SAN shared
storage</description> <link
href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/permissions"
rel="permissions"/>
<link
href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d/disks/16736ce0-a9df-410f-9f29-3a28364cdd41/statistics"
rel="statistics"/>
<alias>hertz_disk5</alias>
<image_id>4ab070c0-fb16-452d-8521-4ff0b004aef3</image_id>
<storage_domain
href="/ovirt-engine/api/storagedomains/7a48fe46-2112-40a4-814f-24d74c760b2d"
id="7a48fe46-2112-40a4-814f-24d74c760b2d"/>
<storage_domains>
<storage_domain id="7a48fe46-2112-40a4-814f-24d74c760b2d"/>
</storage_domains>
<size>225485783040</size>
<provisioned_size>225485783040</provisioned_size>
<actual_size>225485783040</actual_size>
<status>
<state>ok</state>
</status>
<interface>ide</interface>
<format>raw</format>
<sparse>false</sparse>
<bootable>false</bootable>
<shareable>false</shareable>
<wipe_after_delete>false</wipe_after_delete>
<propagate_errors>false</propagate_errors>
</disk>
</disks>
I'm waiting for any comments on the two OVF disk or any advice
before proceeding with the registering command (test in production
environment hmmmmm)
Cheers
AG
-----Original Message-----
From: Andrea Ghelardi [mailto:a.ghelardi@iontrading.com]
Sent: Friday, May 15, 2015 12:30 PM
To: Maor Lipchuk
Cc: users@ovirt.org
Subject: RE: [ovirt-users] missing disk after storage domain
expansion
Thank you for your reply!
I'm unsure if the disk contained any snapshots. I do not think so.
Is the register action a safe one to do in production system? I
wouldn't mess any of my existing running servers.
Here the relevant logs from the date
./engine.log-20150501.gz:2015-04-29 16:10:40,868  : disk creation
("hertz_disk5" newImageId = 4ab070c0-fb16-452d-8521-4ff0b004aef3)
./engine.log-20150512.gz:2015-05-04 17:22:42,691  : last
occurrence of the imageid
Please note from
./engine.log-20150512.gz  2015-05-11 12:07:16,719 : creation of
two disk OVF store (??) on the same storage domain
7a48fe46-2112-40a4-814f-24d74c760b2d
right after the volume expansion
In the instructions.txt  what I did to enlarge the volume (and
lose the
disk) (the guides is informational only, I followed the steps on a
different
server)
Here some cats to thank you in advance
https://www.youtube.com/watch?v=nfTY41-zC7Y
lol :)
Nice cats
...
AG
-----Original Message-----
From: Maor Lipchuk [mailto:mlipchuk@redhat.com]
Sent: Wednesday, May 13, 2015 11:02 PM
To: Andrea Ghelardi
Cc: users@ovirt.org
Subject: Re: [ovirt-users] missing disk after storage domain
expansion
Hi Andrea,
First of all, the issue sounds quite severe, can u please attach
the engine logs so we can try to figure out how that happened.
second, does this disk contained any snapshots?
 If not, can you try to register it back (see
http://www.ovirt.org/Features/ImportStorageDomain#Register_an_unre
gi
st
ered_disk)
Regards,
Maor
----- Original Message -----
...
From: "Andrea Ghelardi" <a.ghelardi@iontrading.com>
To: users@ovirt.org
Sent: Monday, May 11, 2015 12:01:17 AM
Subject: Re: [ovirt-users] missing disk after storage domain
expansion
Ok, so,
After _ a lot _ of unsuccessful approach, I finally connected to
postegre DB directly.
Browsing the tables I found “unregistered_ovf_of_entities” where
there is a reference of the missing disk
@ovf:diskId:4ab070c0-fb16-452d-8521-4ff0b004aef3
@ovf:size:210
@ovf:actual_size:210
@ovf:vm_snapshot_id:2e24b255-bb84-4284-8785-e2a042045882
@ovf:fileRef:16736ce0-a9df-410f-9f29-3a28364cdd41/4ab070c0-fb16-
45
2d
-8
521-4ff0b004aef3
@ovf:format:
http://www.vmware.com/specifications/vmdk.html#sparse
@ovf:volume-format:RAW
@ovf:volume-type:Preallocated
@ovf:disk-interface:VirtIO
@ovf:boot:false
@ovf:disk-alias:hertz_disk5
@ovf:disk-description:disk for SYBASE installation, on SAN
shared storage
@ovf:wipe-after-delete:false
However, I’ve been unable to find any other helpful details.
I guess the disk is not recoverable at this point?
Any guru who has a good ovirt DB kwnoledge willing to give me
some advice?
Thanks as usual
AG
From: Andrea Ghelardi [mailto: a.ghelardi@iontrading.com ]
Sent: Thursday, May 07, 2015 6:08 PM
To: ' users@ovirt.org '
Subject: missing disk after storage domain expansion
Hi gentlemen,
I recently found an error on 1 of my storages: it was
complaining about no free space but the VM was running and disk
operational.
Since I needed to perform some maintenance on the VM, I shut it
down and at restart VM couldn’t boot up properly.
Checked VM via console and a disk was missing. Edited fstab
(luckily this disk was not root but heck! It had a Sybase DB on
it!) and restarted VM this time ok.
Since the disk resides on the dstore with no space, I expanded
the iSCSI LUN, then refreshed multipath on hosts, then resized
PVs and now ovirt is showing the correct size (logs do not
complain anymore on no free space).
BUUUT
Now disk is missing. It is not shown anymore on Disks tab nor
anywhere else.
Problem is that storage shows 214GB occupancy (size of the
missing
disk) so data is there but cannot find it anymore.
Logs show original disk creation, errors from the lack of space,
refresh of the storage size and then.... no more references on
the disk.
What can I do to find those missing ~210GBs?
Cheers
Andrea Ghelardi
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] missing disk after storage domain expansion

Andrea Ghelardi

Maor Lipchuk

Juan Hernández

tags

participants (3)