Re: [Engine-devel] Domain rescan action question

----- Original Message ----- > From: "Ricky Hopper" <Ricky.Hopper(a)netapp.com&gt; > To: "Dan Yasny" <dyasny(a)redhat.com&gt; > Cc: engine-devel(a)ovirt.org, "Itamar Heim" <iheim(a)redhat.com&gt;, "Andrew >Cathrow" <acathrow(a)redhat.com&gt; > Sent: Wednesday, 1 August, 2012 8:36:09 PM > Subject: Re: [Engine-devel] Domain rescan action question > > > > On 8/1/12 10:05 AM, "Dan Yasny" <dyasny(a)redhat.com&gt; wrote: > > > > > > >----- Original Message ----- > >> From: "Ricky Hopper" <Ricky.Hopper(a)netapp.com&gt; > >> To: "Dan Yasny" <dyasny(a)redhat.com&gt; > >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" <iheim(a)redhat.com&gt;, > >> "Andrew > >>Cathrow" <acathrow(a)redhat.com&gt; > >> Sent: Wednesday, 1 August, 2012 4:56:45 PM > >> Subject: Re: [Engine-devel] Domain rescan action question > >> > >> > >> > >> On 8/1/12 9:42 AM, "Dan Yasny" <dyasny(a)redhat.com&gt; wrote: > >> > >> > > >> > > >> >----- Original Message ----- > >> >> From: "Ricky Hopper" <Ricky.Hopper(a)netapp.com&gt; > >> >> To: "Dan Yasny" <dyasny(a)redhat.com&gt;, "Andrew Cathrow" > >> &gt;&gt;&lt;acathrow(a)redhat.com&gt; > >> >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" <iheim(a)redhat.com&gt;, > >> >> "Ricky > >> >>Hopper" <Ricky.Hopper(a)netapp.com&gt; > >> >> Sent: Wednesday, 1 August, 2012 4:34:53 PM > >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> > >> >> > >> >> > >> >> On 8/1/12 5:59 AM, "Dan Yasny" <dyasny(a)redhat.com&gt; wrote: > >> >> > >> >> > > >> >> > > >> >> >----- Original Message ----- > >> >> >> From: "Andrew Cathrow" <acathrow(a)redhat.com&gt; > >> >> >> To: "Itamar Heim" <iheim(a)redhat.com&gt;, "Dan Yasny" > >> >> >> <dyasny(a)redhat.com&gt;, > >> >> >>"Ricky Hopper" <Ricky.Hopper(a)netapp.com&gt; > >> >> >> Cc: engine-devel(a)ovirt.org > >> >> >> Sent: Wednesday, 1 August, 2012 12:24:42 AM > >> >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> >> > >> >> >> > >> >> >> > >> >> >> ----- Original Message ----- > >> >> >> > From: "Itamar Heim" <iheim(a)redhat.com&gt; > >> >> >> > To: "Ricky Hopper" <Ricky.Hopper(a)netapp.com&gt; > >> >> >> > Cc: engine-devel(a)ovirt.org > >> >> >> > Sent: Tuesday, July 31, 2012 4:44:34 PM > >> >> >> > Subject: Re: [Engine-devel] Domain rescan action question > >> >> >> > > >> >> >> > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > >> >> >> > > Hey all, > >> >> >> > > > >> >> >> > > As I'm making progress with the domain rescan > >> >> >> > > functionality, > >> >> >> > > I've > >> >> >> > > realized that I'm unsure what to do with any disks that > >> >> >> > > are > >> >> >> > > detected on > >> >> >> > > the domain. Should I add them back into the database to > >> >> >> > > be > >> >> >> > > listed > >> >> >> > > as > >> >> >> > > floating disks, or should I just return a list of disk > >> >> >> > > images > >> >> >> > > to > >> >> >> > > be > >> >> >> > > attached to whatever the caller of the query needs? > >> >> >> > > > >> >> >> > > - Ricky > >> >> >> > > >> >> >> > i'm not sure they should be added automatically. > >> >> >> > I think a dialog[1] showing orphan disks/images on the > >> >> >> > storage > >> >> >> > domain > >> >> >> > for user to choose which to import as 'floating' disks > >> >> >> > would > >> >> >> > be > >> >> >> > better > >> >> >> > than auto importing them. > >> >> >> > > >> >> >> > there is also the reverse of flagging existing disks as > >> >> >> > 'missing' > >> >> >> > in > >> >> >> > storage? > >> >> >> > > >> >> >> > >> >> >> Perhaps we should start a feature page to discuss and better > >> >> >> scope > >> >> >> it. > >> >> >> There is a feature page that we could expand, it doesn't > >> >> >> discuss > >> >> >> the > >> >> >> notion of importing those disks which is certainly something > >> >> >> we > >> >> >> need > >> >> >> to address. > >> >> >> > >> >> >> > >> >> >> http://wiki.ovirt.org/wiki/Features/Orphaned_Images > >> >> > > >> >> >The original idea was to scan the storage domains and compare > >> >> >the > >> >> >images > >> >> >lists to the database, thus getting a list of images no longer > >> >> >relevant > >> >> >and scrubbing the storage. This will actually be addressed > >> >> >properly > >> >> >in > >> >> >the future (Ayal can elaborate on that) but for now this is > >> >> >needed > >> >> >at > >> >> >least for that use case. > >> >> > > >> >> > > >> >> >As I understand, the conversation here is about trying to take > >> >> >an > >> >> >already > >> >> >populated SD (from another setup I suppose), scanning it and > >> >> >putting > >> >> >it > >> >> >into RHEV? > >> >> > >> >> As I understood it, the purpose of this functionality wasn't to > >> >> find > >> >> images which should be removed from storage, but to find images > >> >> on > >> >> the > >> >> domain that oVirt was unaware of and importing them for use > >> >> (for > >> >> instance, > >> >> if a disk was created outside of oVirt on the domain). If one > >> >> of > >> >> the > >> >> use > >> >> cases for this feature is also the orphaned images mentioned on > >> >> the > >> >> feature page, that may expand the functionality into a separate > >> >> domain > >> >> scrub and storage import, both of which would call the rescan > >> >> (meaning the > >> >> rescan would not actually add to the database, but instead > >> >> return > >> >> a > >> >> list > >> >> of "orphaned" disk images). > >> >> > >> >> Another solution would be to import all disk images into the > >> >> database > >> >> either way, and let the user delete any orphaned images from > >> >> the > >> >> GUI. > >> > > >> >I think are nice to have, but the problem with the scanning is > >> >that > >> >if > >> >we're not scanning a master domain or an export domain, all we > >> >will > >> >see > >> >is a bunch of images with no context or even hints as to where > >> >they > >> >belong. The data that makes it all usable is in the engine > >> >database > >> >and > >> >in the ovf files on the master domain. > >> > > >> >This is why I stopped at the orphaned images part of the feature > >> >- > >> >because there it's feasible, I would rely on the engine database > >> >for > >> >image ID comparisons. > >> > > >> >If we present a user with a list of nameless disks, I doubt it > >> >will > >> >be of > >> >any use. > >> > >> The way this would work is by comparing a list of disk images from > >> vdsm > >> and from oVirt's database, finding the ones vdsm returns that > >> oVirt > >> doesn't have, and then either adding or returning those images. So > >> oVirt's > >> db will be used in the comparison. > > > >This will work only when scanning storage domains already attached > >and in > >use by the current oVirt setup. What I am talking about is what will > >happen if a LUN that used to be a SD in another oVirt setup is > >discovered > >and scanned, with no engine db to compare with. If we don't consider > >such > >a use case, life is definitely quite easy, and we're basically > >within the > >scope of the orphaned images feature > > This use case should definitely be considered, maybe have a separate > case > where the rescan would return all "compatible" disks (i.e. disks that > aren't just partial snapshots and the like) if the domain has not yet > been > mounted. Essentially, it would run the same comparison, but compare > against an empty list rather than a list of disks. There's no way > it's as > simple as that (I'm unsure of the methods oVirt uses to mount a > domain), > but it's a good starting point. There is no complex method there. For file storage it's just a mount command, and for block it's LVM (plus iscsi session establishment, if needed) > > > >> > >> As far as presenting the user with nameless disks, that's a point > >> I > >> hadn't > >> considered; we could generate some sort of placeholder metadata > >> upon > >> addition to show the user that these are new/orphaned disks that > >> were > >> found on the storage domain. Is it safe to assume that the disks > >> discovered by this feature won't be attached to anything? > > > >The oVirt paradigm says "if it isn't in the engine db, it's not > >ours", so > >any LV or image we discover that is missing from the DB or the > >snapshot > >chain of the image in the DB, is nameless, and orphaned. > > > >Such an image on a current SD, belonging to a working oVirt setup is > >definitely an orphaned image. Attaching these to VMs is usually also > >useless, because they are more often than not discarded snapshots > >that > >didn't get discarded cleanly for some reason. > > > > > >Now, if we want to make this usable, we might want to actually check > >the > >qcow2 metadata of the image to see whether it's a mid-chain snapshot > >(and > >if so it's probably just a candidate for cleanup), or a standalone > >qcow2 > >or raw image, and then we can move on with the virt-* tools, to find > >out > >the image size and the filesystems it contains. This will at least > >provide the user with some usable information about the detected > >image. > >If we're talking about scanning an SD that doesn't presently belong > >to > >the current oVirt setup, then this is even more relevant, because > >all of > >the images will have no VM-related context. > > We're currently working on having disks created outside of the oVirt > environment, so not all orphaned disks on the existing storage domain > will > be artifacts of supposedly-deleted data. Do you mean like rhevm-image-upload, or something different? > For our use case, disk > images > created by us will be able to be imported into oVirt and attached to > a VM > created through the engine. Because of this, saying "if it isn't in > the > engine db, it's not ours" wouldn't necessarily be true. > > When you talk about checking the metadata, does either oVirt or vdsm > have > a simple way to do this? A query of some sort would be ideal for > this, as > it could be run for each image as a qualifier for import. qemu-img info and libguestfs commands should do. Besides, our images do come with some metadata (in the LVM tags or a .meta file) > > Also, as far as writing the functionality itself, I'm gathering that > it > should be structured as a query to return these orphaned images, > which can > then be acted upon/added to the database through a separate command > after > checking the validity of each image? Yes, a simple way to say "import this one to DB, attach to VM X or make floating", "delete that one", "skip" > > > >> > > >> > > >> >> > > >> >> >> > >> >> >> > > >> >> >> > [1] or a subtab on the storage domain. > >> >> >> > > >> >> >> > _______________________________________________ > >> >> >> > Engine-devel mailing list > >> >> >> > Engine-devel(a)ovirt.org > >> >> >> > http://lists.ovirt.org/mailman/listinfo/engine-devel > >> >> >> > > >> >> >> > >> >> > > >> >> >-- > >> >> > > >> >> > > >> >> > > >> >> >Regards, > >> >> > > >> >> >Dan Yasny > >> >> >Red Hat Israel > >> >> >+972 9769 2280 > >> >> > >> >> > >> > > >> >-- > >> > > >> > > >> > > >> >Regards, > >> > > >> >Dan Yasny > >> >Red Hat Israel > >> >+972 9769 2280 > >> > >> > > > >-- > > > > > > > >Regards, > > > >Dan Yasny > >Red Hat Israel > >+972 9769 2280 > > -- Regards, Dan Yasny Red Hat Israel +972 9769 2280 _______________________________________________ Engine-devel mailing list Engine-devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel