As I'm making progress with the domain rescan functionality, I've real= =3D ized that I'm unsure what to do with any disks that are detected on the dom= =3D ain. Should I add them back into the database to be listed as floating disk= =3D s, or should I just return a list of disk images to be attached to whatever the caller of the query needs?

- Ricky

--_000_CC3DB8AB143Crickyhnetappcom_-- --===============3258282432404943241== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX0NDM0RCOEFCMTQzQ3JpY2t5aG5ldGFwcGNvbV8KQ29udGVudC1UeXBlOiB0ZXh0L3Bs YWluOyBjaGFyc2V0PSJ1cy1hc2NpaSIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVk LXByaW50YWJsZQoKSGV5IGFsbCwKCkFzIEknbSBtYWtpbmcgcHJvZ3Jlc3Mgd2l0aCB0aGUgZG9t YWluIHJlc2NhbiBmdW5jdGlvbmFsaXR5LCBJJ3ZlIHJlYWxpemVkID0KdGhhdCBJJ20gdW5zdXJl IHdoYXQgdG8gZG8gd2l0aCBhbnkgZGlza3MgdGhhdCBhcmUgZGV0ZWN0ZWQgb24gdGhlIGRvbWFp bi4gPQpTaG91bGQgSSBhZGQgdGhlbSBiYWNrIGludG8gdGhlIGRhdGFiYXNlIHRvIGJlIGxpc3Rl ZCBhcyBmbG9hdGluZyBkaXNrcywgb3I9CiBzaG91bGQgSSBqdXN0IHJldHVybiBhIGxpc3Qgb2Yg ZGlzayBpbWFnZXMgdG8gYmUgYXR0YWNoZWQgdG8gd2hhdGV2ZXIgdGhlID0KY2FsbGVyIG9mIHRo ZSBxdWVyeSBuZWVkcz8KCi0gUmlja3kKCi0tXzAwMF9DQzNEQjhBQjE0M0NyaWNreWhuZXRhcHBj b21fCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PSJ1cy1hc2NpaSIKQ29udGVudC1J RDogPEFGRUQxOEI3MjRCRUVFNDY4MzI0NTI1Mzk3NkNCRjdCQHRhaG9lLm5ldGFwcC5jb20+CkNv bnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKCjxodG1sPgo8aGVhZD4K PG1ldGEgaHR0cC1lcXVpdj0zRCJDb250ZW50LVR5cGUiIGNvbnRlbnQ9M0QidGV4dC9odG1sOyBj aGFyc2V0PTNEdXMtYXNjaWkiPQo+CjwvaGVhZD4KPGJvZHkgc3R5bGU9M0Qid29yZC13cmFwOiBi cmVhay13b3JkOyAtd2Via2l0LW5ic3AtbW9kZTogc3BhY2U7IC13ZWJraXQtbGluPQplLWJyZWFr OiBhZnRlci13aGl0ZS1zcGFjZTsgY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1zaXplOiAxNHB4 OyBmb250LWZhbWk9Cmx5OiBDYWxpYnJpLCBzYW5zLXNlcmlmOyAiPgo8ZGl2PkhleSBhbGwsPC9k aXY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5BcyBJJ20gbWFraW5nIHByb2dyZXNzIHdpdGggdGhl IGRvbWFpbiByZXNjYW4gZnVuY3Rpb25hbGl0eSwgSSd2ZSByZWFsPQppemVkIHRoYXQgSSdtIHVu c3VyZSB3aGF0IHRvIGRvIHdpdGggYW55IGRpc2tzIHRoYXQgYXJlIGRldGVjdGVkIG9uIHRoZSBk b209CmFpbi4gU2hvdWxkIEkgYWRkIHRoZW0gYmFjayBpbnRvIHRoZSBkYXRhYmFzZSB0byBiZSBs aXN0ZWQgYXMgZmxvYXRpbmcgZGlzaz0Kcywgb3Igc2hvdWxkIEkganVzdCByZXR1cm4gYSBsaXN0 CiBvZiBkaXNrIGltYWdlcyB0byBiZSBhdHRhY2hlZCB0byB3aGF0ZXZlciB0aGUgY2FsbGVyIG9m IHRoZSBxdWVyeSBuZWVkcz88Lz0KZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+LSBSaWNreTwv ZGl2Pgo8L2JvZHk+CjwvaHRtbD4KCi0tXzAwMF9DQzNEQjhBQjE0M0NyaWNreWhuZXRhcHBjb21f LS0K --===============3258282432404943241==-- From iheim at redhat.com Tue Jul 31 17:20:50 2012 Content-Type: multipart/mixed; boundary="===============2781942844148036709==" MIME-Version: 1.0 From: Itamar Heim To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Tue, 31 Jul 2012 23:44:34 +0300 Message-ID: <501843B2.7030701@redhat.com> In-Reply-To: CC3DB8AB.143C%rickyh@netapp.com --===============2781942844148036709== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > Hey all, > > As I'm making progress with the domain rescan functionality, I've > realized that I'm unsure what to do with any disks that are detected on > the domain. Should I add them back into the database to be listed as > floating disks, or should I just return a list of disk images to be > attached to whatever the caller of the query needs? > > - Ricky i'm not sure they should be added automatically. I think a dialog[1] showing orphan disks/images on the storage domain = for user to choose which to import as 'floating' disks would be better = than auto importing them. there is also the reverse of flagging existing disks as 'missing' in = storage? [1] or a subtab on the storage domain. --===============2781942844148036709==-- From abaron at redhat.com Tue Jul 31 17:21:59 2012 Content-Type: multipart/mixed; boundary="===============3635931112299640888==" MIME-Version: 1.0 From: Ayal Baron To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Tue, 31 Jul 2012 17:21:58 -0400 Message-ID: <1401554098.6341190.1343769718592.JavaMail.root@redhat.com> In-Reply-To: 501843B2.7030701@redhat.com --===============3635931112299640888== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----- Original Message ----- > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > > Hey all, > > > > As I'm making progress with the domain rescan functionality, I've > > realized that I'm unsure what to do with any disks that are > > detected on > > the domain. Should I add them back into the database to be listed > > as > > floating disks, or should I just return a list of disk images to be > > attached to whatever the caller of the query needs? > > > > - Ricky > = > i'm not sure they should be added automatically. > I think a dialog[1] showing orphan disks/images on the storage domain > for user to choose which to import as 'floating' disks would be > better > than auto importing them. why? this same functionality would be used to import an existing domain. If these disks are referenced in OVFs we are not familiar with on this doma= in then we should import the *VMs*. If they are referenced by other VMs that are already in the system (but dis= ks have been unreachable until now) then the disks should just be added to = the db in attached mode. If neither, then the disks should be added as floating disks. For the import functionality, once you subsequently import another domain w= ith OVFs which do reference these disks then if user hasn't appropriated th= em for other VMs then they would move to attached state, otherwise need to = add those VMs with errors. Note that there are 2 reasons for unknown valid disks to be on the domain: 1. delete was initiated in engine but not performed on storage (then floati= ng is fine as only way to automatically delete them is if user chooses to d= o so) 2. disks were created there outside of the system - should just detect and = import and use logic above. > = > there is also the reverse of flagging existing disks as 'missing' in > storage? If disks were floating then they should just be removed, otherwise should b= e moved to illegal state (we have this state for disks today). > = > = > [1] or a subtab on the storage domain. = Another sub-tab for disks? It's possible but what would you do when importing an existing domain into = the system? require user to manually select which disks to import? or would= you have different flows for import of domain and rescan of contents? (I'd= rather keep it simple with less tabs and manual operations for user to per= form, seems more intuitive to me). --===============3635931112299640888==-- From acathrow at redhat.com Tue Jul 31 17:24:43 2012 Content-Type: multipart/mixed; boundary="===============0867559705584849048==" MIME-Version: 1.0 From: Andrew Cathrow To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Tue, 31 Jul 2012 17:24:42 -0400 Message-ID: <1049657803.1388688.1343769882067.JavaMail.root@redhat.com> In-Reply-To: 501843B2.7030701@redhat.com --===============0867559705584849048== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----- Original Message ----- > From: "Itamar Heim" > To: "Ricky Hopper" > Cc: engine-devel(a)ovirt.org > Sent: Tuesday, July 31, 2012 4:44:34 PM > Subject: Re: [Engine-devel] Domain rescan action question > = > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > > Hey all, > > > > As I'm making progress with the domain rescan functionality, I've > > realized that I'm unsure what to do with any disks that are > > detected on > > the domain. Should I add them back into the database to be listed > > as > > floating disks, or should I just return a list of disk images to be > > attached to whatever the caller of the query needs? > > > > - Ricky > = > i'm not sure they should be added automatically. > I think a dialog[1] showing orphan disks/images on the storage domain > for user to choose which to import as 'floating' disks would be > better > than auto importing them. > = > there is also the reverse of flagging existing disks as 'missing' in > storage? > = Perhaps we should start a feature page to discuss and better scope it. There is a feature page that we could expand, it doesn't discuss the notion= of importing those disks which is certainly something we need to address. http://wiki.ovirt.org/wiki/Features/Orphaned_Images > = > [1] or a subtab on the storage domain. > = > _______________________________________________ > Engine-devel mailing list > Engine-devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel >=20 --===============0867559705584849048==-- From iheim at redhat.com Wed Aug 1 00:45:15 2012 Content-Type: multipart/mixed; boundary="===============9154494105755630367==" MIME-Version: 1.0 From: Itamar Heim To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Wed, 01 Aug 2012 07:45:10 +0300 Message-ID: <5018B456.8030700@redhat.com> In-Reply-To: 1401554098.6341190.1343769718592.JavaMail.root@redhat.com --===============9154494105755630367== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 08/01/2012 12:21 AM, Ayal Baron wrote: > > > ----- Original Message ----- >> On 07/31/2012 11:30 PM, Hopper, Ricky wrote: >>> Hey all, >>> >>> As I'm making progress with the domain rescan functionality, I've >>> realized that I'm unsure what to do with any disks that are >>> detected on >>> the domain. Should I add them back into the database to be listed >>> as >>> floating disks, or should I just return a list of disk images to be >>> attached to whatever the caller of the query needs? >>> >>> - Ricky >> >> i'm not sure they should be added automatically. >> I think a dialog[1] showing orphan disks/images on the storage domain >> for user to choose which to import as 'floating' disks would be >> better >> than auto importing them. > > why? this same functionality would be used to import an existing domain. > If these disks are referenced in OVFs we are not familiar with on this do= main then we should import the *VMs*. > If they are referenced by other VMs that are already in the system (but d= isks have been unreachable until now) then the disks should just be added t= o the db in attached mode. > If neither, then the disks should be added as floating disks. > For the import functionality, once you subsequently import another domain= with OVFs which do reference these disks then if user hasn't appropriated = them for other VMs then they would move to attached state, otherwise need t= o add those VMs with errors. > > Note that there are 2 reasons for unknown valid disks to be on the domain: > 1. delete was initiated in engine but not performed on storage (then floa= ting is fine as only way to automatically delete them is if user chooses to= do so) > 2. disks were created there outside of the system - should just detect an= d import and use logic above. > >> >> there is also the reverse of flagging existing disks as 'missing' in >> storage? > > If disks were floating then they should just be removed, otherwise should= be moved to illegal state (we have this state for disks today). > >> >> >> [1] or a subtab on the storage domain. > > Another sub-tab for disks? > It's possible but what would you do when importing an existing domain int= o the system? require user to manually select which disks to import? or wou= ld you have different flows for import of domain and rescan of contents? (I= 'd rather keep it simple with less tabs and manual operations for user to p= erform, seems more intuitive to me). > I'm not sure orphaed images and importing an entire domain should be the = same flow from UX part. and for a netapp native clone, you'd want to only import the newly = cloned ("orphaned) image from the storage via an api call, not the = entire domain/images --===============9154494105755630367==-- From abaron at redhat.com Wed Aug 1 03:48:30 2012 Content-Type: multipart/mixed; boundary="===============1785738560909950099==" MIME-Version: 1.0 From: Ayal Baron To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Wed, 01 Aug 2012 03:48:28 -0400 Message-ID: <571010747.6484571.1343807308605.JavaMail.root@redhat.com> In-Reply-To: 5018B456.8030700@redhat.com --===============1785738560909950099== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----- Original Message ----- > On 08/01/2012 12:21 AM, Ayal Baron wrote: > > > > > > ----- Original Message ----- > >> On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > >>> Hey all, > >>> > >>> As I'm making progress with the domain rescan functionality, I've > >>> realized that I'm unsure what to do with any disks that are > >>> detected on > >>> the domain. Should I add them back into the database to be listed > >>> as > >>> floating disks, or should I just return a list of disk images to > >>> be > >>> attached to whatever the caller of the query needs? > >>> > >>> - Ricky > >> > >> i'm not sure they should be added automatically. > >> I think a dialog[1] showing orphan disks/images on the storage > >> domain > >> for user to choose which to import as 'floating' disks would be > >> better > >> than auto importing them. > > > > why? this same functionality would be used to import an existing > > domain. > > If these disks are referenced in OVFs we are not familiar with on > > this domain then we should import the *VMs*. > > If they are referenced by other VMs that are already in the system > > (but disks have been unreachable until now) then the disks should > > just be added to the db in attached mode. > > If neither, then the disks should be added as floating disks. > > For the import functionality, once you subsequently import another > > domain with OVFs which do reference these disks then if user > > hasn't appropriated them for other VMs then they would move to > > attached state, otherwise need to add those VMs with errors. > > > > Note that there are 2 reasons for unknown valid disks to be on the > > domain: > > 1. delete was initiated in engine but not performed on storage > > (then floating is fine as only way to automatically delete them is > > if user chooses to do so) > > 2. disks were created there outside of the system - should just > > detect and import and use logic above. > > > >> > >> there is also the reverse of flagging existing disks as 'missing' > >> in > >> storage? > > > > If disks were floating then they should just be removed, otherwise > > should be moved to illegal state (we have this state for disks > > today). > > > >> > >> > >> [1] or a subtab on the storage domain. > > > > Another sub-tab for disks? > > It's possible but what would you do when importing an existing > > domain into the system? require user to manually select which > > disks to import? or would you have different flows for import of > > domain and rescan of contents? (I'd rather keep it simple with > > less tabs and manual operations for user to perform, seems more > > intuitive to me). > > > = > I'm not sure orphaed images and importing an entire domain should be > the > same flow from UX part. > and for a netapp native clone, you'd want to only import the newly > cloned ("orphaned) image from the storage via an api call, not the > entire domain/images For proper bookkeeping I think we'd want these disks to be reflected in the= GUI. I do not however think we need to add additional tabs for this. We could have yet another state ('orphaned' or sth.) for disks to be able t= o easily differentiate in the GUI etc. I would imagine we'd regularly scan domains to account for storage usage. If we do add such a state then we'd need an API to 'enable' it (so that it = could be done automatically). =20 --===============1785738560909950099==-- From ofrenkel at redhat.com Wed Aug 1 05:43:28 2012 Content-Type: multipart/mixed; boundary="===============6073299349078464354==" MIME-Version: 1.0 From: Omer Frenkel To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Wed, 01 Aug 2012 05:43:27 -0400 Message-ID: <253410604.1614860.1343814207457.JavaMail.root@redhat.com> In-Reply-To: 1049657803.1388688.1343769882067.JavaMail.root@redhat.com --===============6073299349078464354== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----- Original Message ----- > From: "Andrew Cathrow" > To: "Itamar Heim" , "Dan Yasny" = , "Ricky Hopper" > Cc: engine-devel(a)ovirt.org > Sent: Wednesday, August 1, 2012 12:24:42 AM > Subject: Re: [Engine-devel] Domain rescan action question > = > = > = > ----- Original Message ----- > > From: "Itamar Heim" > > To: "Ricky Hopper" > > Cc: engine-devel(a)ovirt.org > > Sent: Tuesday, July 31, 2012 4:44:34 PM > > Subject: Re: [Engine-devel] Domain rescan action question > > = > > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > > > Hey all, > > > > > > As I'm making progress with the domain rescan functionality, I've > > > realized that I'm unsure what to do with any disks that are > > > detected on > > > the domain. Should I add them back into the database to be listed > > > as > > > floating disks, or should I just return a list of disk images to > > > be > > > attached to whatever the caller of the query needs? > > > > > > - Ricky > > = > > i'm not sure they should be added automatically. > > I think a dialog[1] showing orphan disks/images on the storage > > domain > > for user to choose which to import as 'floating' disks would be > > better > > than auto importing them. > > = > > there is also the reverse of flagging existing disks as 'missing' > > in > > storage? > > = > = > Perhaps we should start a feature page to discuss and better scope > it. > There is a feature page that we could expand, it doesn't discuss the > notion of importing those disks which is certainly something we need > to address. > = > = > http://wiki.ovirt.org/wiki/Features/Orphaned_Images > = +1 also, how will be handled images with snapshots? or broken chain of images (not sure its a valid scenario) > > = > > [1] or a subtab on the storage domain. > > = > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > = > _______________________________________________ > Engine-devel mailing list > Engine-devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel >=20 --===============6073299349078464354==-- From dyasny at redhat.com Wed Aug 1 05:59:12 2012 Content-Type: multipart/mixed; boundary="===============5109387434973388759==" MIME-Version: 1.0 From: Dan Yasny To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Wed, 01 Aug 2012 05:59:10 -0400 Message-ID: <1631790052.4861341.1343815150755.JavaMail.root@redhat.com> In-Reply-To: 1049657803.1388688.1343769882067.JavaMail.root@redhat.com --===============5109387434973388759== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----- Original Message ----- > From: "Andrew Cathrow" > To: "Itamar Heim" , "Dan Yasny" = , "Ricky Hopper" > Cc: engine-devel(a)ovirt.org > Sent: Wednesday, 1 August, 2012 12:24:42 AM > Subject: Re: [Engine-devel] Domain rescan action question > = > = > = > ----- Original Message ----- > > From: "Itamar Heim" > > To: "Ricky Hopper" > > Cc: engine-devel(a)ovirt.org > > Sent: Tuesday, July 31, 2012 4:44:34 PM > > Subject: Re: [Engine-devel] Domain rescan action question > > = > > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > > > Hey all, > > > > > > As I'm making progress with the domain rescan functionality, I've > > > realized that I'm unsure what to do with any disks that are > > > detected on > > > the domain. Should I add them back into the database to be listed > > > as > > > floating disks, or should I just return a list of disk images to > > > be > > > attached to whatever the caller of the query needs? > > > > > > - Ricky > > = > > i'm not sure they should be added automatically. > > I think a dialog[1] showing orphan disks/images on the storage > > domain > > for user to choose which to import as 'floating' disks would be > > better > > than auto importing them. > > = > > there is also the reverse of flagging existing disks as 'missing' > > in > > storage? > > = > = > Perhaps we should start a feature page to discuss and better scope > it. > There is a feature page that we could expand, it doesn't discuss the > notion of importing those disks which is certainly something we need > to address. > = > = > http://wiki.ovirt.org/wiki/Features/Orphaned_Images The original idea was to scan the storage domains and compare the images li= sts to the database, thus getting a list of images no longer relevant and s= crubbing the storage. This will actually be addressed properly in the futur= e (Ayal can elaborate on that) but for now this is needed at least for that= use case. As I understand, the conversation here is about trying to take an already p= opulated SD (from another setup I suppose), scanning it and putting it into= RHEV? > = > > = > > [1] or a subtab on the storage domain. > > = > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > = > = -- = Regards, = Dan Yasny = Red Hat Israel = +972 9769 2280 --===============5109387434973388759==-- From Ricky.Hopper at netapp.com Wed Aug 1 09:35:21 2012 Content-Type: multipart/mixed; boundary="===============7345584901711360291==" MIME-Version: 1.0 From: Hopper, Ricky To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Wed, 01 Aug 2012 13:34:53 +0000 Message-ID: In-Reply-To: 1631790052.4861341.1343815150755.JavaMail.root@redhat.com --===============7345584901711360291== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 8/1/12 5:59 AM, "Dan Yasny" wrote: > > >----- Original Message ----- >> From: "Andrew Cathrow" >> To: "Itamar Heim" , "Dan Yasny" , >>"Ricky Hopper" >> Cc: engine-devel(a)ovirt.org >> Sent: Wednesday, 1 August, 2012 12:24:42 AM >> Subject: Re: [Engine-devel] Domain rescan action question >> = >> = >> = >> ----- Original Message ----- >> > From: "Itamar Heim" >> > To: "Ricky Hopper" >> > Cc: engine-devel(a)ovirt.org >> > Sent: Tuesday, July 31, 2012 4:44:34 PM >> > Subject: Re: [Engine-devel] Domain rescan action question >> > = >> > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: >> > > Hey all, >> > > >> > > As I'm making progress with the domain rescan functionality, I've >> > > realized that I'm unsure what to do with any disks that are >> > > detected on >> > > the domain. Should I add them back into the database to be listed >> > > as >> > > floating disks, or should I just return a list of disk images to >> > > be >> > > attached to whatever the caller of the query needs? >> > > >> > > - Ricky >> > = >> > i'm not sure they should be added automatically. >> > I think a dialog[1] showing orphan disks/images on the storage >> > domain >> > for user to choose which to import as 'floating' disks would be >> > better >> > than auto importing them. >> > = >> > there is also the reverse of flagging existing disks as 'missing' >> > in >> > storage? >> > = >> = >> Perhaps we should start a feature page to discuss and better scope >> it. >> There is a feature page that we could expand, it doesn't discuss the >> notion of importing those disks which is certainly something we need >> to address. >> = >> = >> http://wiki.ovirt.org/wiki/Features/Orphaned_Images > >The original idea was to scan the storage domains and compare the images >lists to the database, thus getting a list of images no longer relevant >and scrubbing the storage. This will actually be addressed properly in >the future (Ayal can elaborate on that) but for now this is needed at >least for that use case. > > >As I understand, the conversation here is about trying to take an already >populated SD (from another setup I suppose), scanning it and putting it >into RHEV? As I understood it, the purpose of this functionality wasn't to find images which should be removed from storage, but to find images on the domain that oVirt was unaware of and importing them for use (for instance, if a disk was created outside of oVirt on the domain). If one of the use cases for this feature is also the orphaned images mentioned on the feature page, that may expand the functionality into a separate domain scrub and storage import, both of which would call the rescan (meaning the rescan would not actually add to the database, but instead return a list of "orphaned" disk images). Another solution would be to import all disk images into the database either way, and let the user delete any orphaned images from the GUI. > >> = >> > = >> > [1] or a subtab on the storage domain. >> > = >> > _______________________________________________ >> > Engine-devel mailing list >> > Engine-devel(a)ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/engine-devel >> > = >> = > >-- = > > > >Regards, = > >Dan Yasny = >Red Hat Israel = >+972 9769 2280 --===============7345584901711360291==-- From dyasny at redhat.com Wed Aug 1 09:42:38 2012 Content-Type: multipart/mixed; boundary="===============7675306676379573508==" MIME-Version: 1.0 From: Dan Yasny To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Wed, 01 Aug 2012 09:42:37 -0400 Message-ID: <176087497.4951388.1343828557474.JavaMail.root@redhat.com> In-Reply-To: CC3EA656.1473%rickyh@netapp.com --===============7675306676379573508== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----- Original Message ----- > From: "Ricky Hopper" > To: "Dan Yasny" , "Andrew Cathrow" > Cc: engine-devel(a)ovirt.org, "Itamar Heim" , "Ricky = Hopper" > Sent: Wednesday, 1 August, 2012 4:34:53 PM > Subject: Re: [Engine-devel] Domain rescan action question > = > = > = > On 8/1/12 5:59 AM, "Dan Yasny" wrote: > = > > > > > >----- Original Message ----- > >> From: "Andrew Cathrow" > >> To: "Itamar Heim" , "Dan Yasny" > >> , > >>"Ricky Hopper" > >> Cc: engine-devel(a)ovirt.org > >> Sent: Wednesday, 1 August, 2012 12:24:42 AM > >> Subject: Re: [Engine-devel] Domain rescan action question > >> = > >> = > >> = > >> ----- Original Message ----- > >> > From: "Itamar Heim" > >> > To: "Ricky Hopper" > >> > Cc: engine-devel(a)ovirt.org > >> > Sent: Tuesday, July 31, 2012 4:44:34 PM > >> > Subject: Re: [Engine-devel] Domain rescan action question > >> > = > >> > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > >> > > Hey all, > >> > > > >> > > As I'm making progress with the domain rescan functionality, > >> > > I've > >> > > realized that I'm unsure what to do with any disks that are > >> > > detected on > >> > > the domain. Should I add them back into the database to be > >> > > listed > >> > > as > >> > > floating disks, or should I just return a list of disk images > >> > > to > >> > > be > >> > > attached to whatever the caller of the query needs? > >> > > > >> > > - Ricky > >> > = > >> > i'm not sure they should be added automatically. > >> > I think a dialog[1] showing orphan disks/images on the storage > >> > domain > >> > for user to choose which to import as 'floating' disks would be > >> > better > >> > than auto importing them. > >> > = > >> > there is also the reverse of flagging existing disks as > >> > 'missing' > >> > in > >> > storage? > >> > = > >> = > >> Perhaps we should start a feature page to discuss and better scope > >> it. > >> There is a feature page that we could expand, it doesn't discuss > >> the > >> notion of importing those disks which is certainly something we > >> need > >> to address. > >> = > >> = > >> http://wiki.ovirt.org/wiki/Features/Orphaned_Images > > > >The original idea was to scan the storage domains and compare the > >images > >lists to the database, thus getting a list of images no longer > >relevant > >and scrubbing the storage. This will actually be addressed properly > >in > >the future (Ayal can elaborate on that) but for now this is needed > >at > >least for that use case. > > > > > >As I understand, the conversation here is about trying to take an > >already > >populated SD (from another setup I suppose), scanning it and putting > >it > >into RHEV? > = > As I understood it, the purpose of this functionality wasn't to find > images which should be removed from storage, but to find images on > the > domain that oVirt was unaware of and importing them for use (for > instance, > if a disk was created outside of oVirt on the domain). If one of the > use > cases for this feature is also the orphaned images mentioned on the > feature page, that may expand the functionality into a separate > domain > scrub and storage import, both of which would call the rescan > (meaning the > rescan would not actually add to the database, but instead return a > list > of "orphaned" disk images). > = > Another solution would be to import all disk images into the database > either way, and let the user delete any orphaned images from the GUI. I think are nice to have, but the problem with the scanning is that if we'r= e not scanning a master domain or an export domain, all we will see is a bu= nch of images with no context or even hints as to where they belong. The da= ta that makes it all usable is in the engine database and in the ovf files = on the master domain. = This is why I stopped at the orphaned images part of the feature - because = there it's feasible, I would rely on the engine database for image ID compa= risons. = If we present a user with a list of nameless disks, I doubt it will be of a= ny use. = > > > >> = > >> > = > >> > [1] or a subtab on the storage domain. > >> > = > >> > _______________________________________________ > >> > Engine-devel mailing list > >> > Engine-devel(a)ovirt.org > >> > http://lists.ovirt.org/mailman/listinfo/engine-devel > >> > = > >> = > > > >-- > > > > > > > >Regards, > > > >Dan Yasny > >Red Hat Israel > >+972 9769 2280 > = > = -- = Regards, = Dan Yasny = Red Hat Israel = +972 9769 2280 --===============7675306676379573508==-- From Ricky.Hopper at netapp.com Wed Aug 1 09:56:47 2012 Content-Type: multipart/mixed; boundary="===============4426522851859452747==" MIME-Version: 1.0 From: Hopper, Ricky To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Wed, 01 Aug 2012 13:56:45 +0000 Message-ID: In-Reply-To: 176087497.4951388.1343828557474.JavaMail.root@redhat.com --===============4426522851859452747== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 8/1/12 9:42 AM, "Dan Yasny" wrote: > > >----- Original Message ----- >> From: "Ricky Hopper" >> To: "Dan Yasny" , "Andrew Cathrow" >> >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , "Ricky >>Hopper" >> Sent: Wednesday, 1 August, 2012 4:34:53 PM >> Subject: Re: [Engine-devel] Domain rescan action question >> = >> = >> = >> On 8/1/12 5:59 AM, "Dan Yasny" wrote: >> = >> > >> > >> >----- Original Message ----- >> >> From: "Andrew Cathrow" >> >> To: "Itamar Heim" , "Dan Yasny" >> >> , >> >>"Ricky Hopper" >> >> Cc: engine-devel(a)ovirt.org >> >> Sent: Wednesday, 1 August, 2012 12:24:42 AM >> >> Subject: Re: [Engine-devel] Domain rescan action question >> >> = >> >> = >> >> = >> >> ----- Original Message ----- >> >> > From: "Itamar Heim" >> >> > To: "Ricky Hopper" >> >> > Cc: engine-devel(a)ovirt.org >> >> > Sent: Tuesday, July 31, 2012 4:44:34 PM >> >> > Subject: Re: [Engine-devel] Domain rescan action question >> >> > = >> >> > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: >> >> > > Hey all, >> >> > > >> >> > > As I'm making progress with the domain rescan functionality, >> >> > > I've >> >> > > realized that I'm unsure what to do with any disks that are >> >> > > detected on >> >> > > the domain. Should I add them back into the database to be >> >> > > listed >> >> > > as >> >> > > floating disks, or should I just return a list of disk images >> >> > > to >> >> > > be >> >> > > attached to whatever the caller of the query needs? >> >> > > >> >> > > - Ricky >> >> > = >> >> > i'm not sure they should be added automatically. >> >> > I think a dialog[1] showing orphan disks/images on the storage >> >> > domain >> >> > for user to choose which to import as 'floating' disks would be >> >> > better >> >> > than auto importing them. >> >> > = >> >> > there is also the reverse of flagging existing disks as >> >> > 'missing' >> >> > in >> >> > storage? >> >> > = >> >> = >> >> Perhaps we should start a feature page to discuss and better scope >> >> it. >> >> There is a feature page that we could expand, it doesn't discuss >> >> the >> >> notion of importing those disks which is certainly something we >> >> need >> >> to address. >> >> = >> >> = >> >> http://wiki.ovirt.org/wiki/Features/Orphaned_Images >> > >> >The original idea was to scan the storage domains and compare the >> >images >> >lists to the database, thus getting a list of images no longer >> >relevant >> >and scrubbing the storage. This will actually be addressed properly >> >in >> >the future (Ayal can elaborate on that) but for now this is needed >> >at >> >least for that use case. >> > >> > >> >As I understand, the conversation here is about trying to take an >> >already >> >populated SD (from another setup I suppose), scanning it and putting >> >it >> >into RHEV? >> = >> As I understood it, the purpose of this functionality wasn't to find >> images which should be removed from storage, but to find images on >> the >> domain that oVirt was unaware of and importing them for use (for >> instance, >> if a disk was created outside of oVirt on the domain). If one of the >> use >> cases for this feature is also the orphaned images mentioned on the >> feature page, that may expand the functionality into a separate >> domain >> scrub and storage import, both of which would call the rescan >> (meaning the >> rescan would not actually add to the database, but instead return a >> list >> of "orphaned" disk images). >> = >> Another solution would be to import all disk images into the database >> either way, and let the user delete any orphaned images from the GUI. > >I think are nice to have, but the problem with the scanning is that if >we're not scanning a master domain or an export domain, all we will see >is a bunch of images with no context or even hints as to where they >belong. The data that makes it all usable is in the engine database and >in the ovf files on the master domain. > >This is why I stopped at the orphaned images part of the feature - >because there it's feasible, I would rely on the engine database for >image ID comparisons. > >If we present a user with a list of nameless disks, I doubt it will be of >any use. The way this would work is by comparing a list of disk images from vdsm and from oVirt's database, finding the ones vdsm returns that oVirt doesn't have, and then either adding or returning those images. So oVirt's db will be used in the comparison. As far as presenting the user with nameless disks, that's a point I hadn't considered; we could generate some sort of placeholder metadata upon addition to show the user that these are new/orphaned disks that were found on the storage domain. Is it safe to assume that the disks discovered by this feature won't be attached to anything? > = > >> > >> >> = >> >> > = >> >> > [1] or a subtab on the storage domain. >> >> > = >> >> > _______________________________________________ >> >> > Engine-devel mailing list >> >> > Engine-devel(a)ovirt.org >> >> > http://lists.ovirt.org/mailman/listinfo/engine-devel >> >> > = >> >> = >> > >> >-- >> > >> > >> > >> >Regards, >> > >> >Dan Yasny >> >Red Hat Israel >> >+972 9769 2280 >> = >> = > >-- = > > > >Regards, = > >Dan Yasny = >Red Hat Israel = >+972 9769 2280 --===============4426522851859452747==-- From dyasny at redhat.com Wed Aug 1 10:05:17 2012 Content-Type: multipart/mixed; boundary="===============4411213061235486299==" MIME-Version: 1.0 From: Dan Yasny To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Wed, 01 Aug 2012 10:05:15 -0400 Message-ID: <1235552045.4962754.1343829915979.JavaMail.root@redhat.com> In-Reply-To: CC3EAB6A.148C%rickyh@netapp.com --===============4411213061235486299== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----- Original Message ----- > From: "Ricky Hopper" > To: "Dan Yasny" > Cc: engine-devel(a)ovirt.org, "Itamar Heim" , "Andrew= Cathrow" > Sent: Wednesday, 1 August, 2012 4:56:45 PM > Subject: Re: [Engine-devel] Domain rescan action question > = > = > = > On 8/1/12 9:42 AM, "Dan Yasny" wrote: > = > > > > > >----- Original Message ----- > >> From: "Ricky Hopper" > >> To: "Dan Yasny" , "Andrew Cathrow" > >> > >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , > >> "Ricky > >>Hopper" > >> Sent: Wednesday, 1 August, 2012 4:34:53 PM > >> Subject: Re: [Engine-devel] Domain rescan action question > >> = > >> = > >> = > >> On 8/1/12 5:59 AM, "Dan Yasny" wrote: > >> = > >> > > >> > > >> >----- Original Message ----- > >> >> From: "Andrew Cathrow" > >> >> To: "Itamar Heim" , "Dan Yasny" > >> >> , > >> >>"Ricky Hopper" > >> >> Cc: engine-devel(a)ovirt.org > >> >> Sent: Wednesday, 1 August, 2012 12:24:42 AM > >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> = > >> >> = > >> >> = > >> >> ----- Original Message ----- > >> >> > From: "Itamar Heim" > >> >> > To: "Ricky Hopper" > >> >> > Cc: engine-devel(a)ovirt.org > >> >> > Sent: Tuesday, July 31, 2012 4:44:34 PM > >> >> > Subject: Re: [Engine-devel] Domain rescan action question > >> >> > = > >> >> > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > >> >> > > Hey all, > >> >> > > > >> >> > > As I'm making progress with the domain rescan > >> >> > > functionality, > >> >> > > I've > >> >> > > realized that I'm unsure what to do with any disks that are > >> >> > > detected on > >> >> > > the domain. Should I add them back into the database to be > >> >> > > listed > >> >> > > as > >> >> > > floating disks, or should I just return a list of disk > >> >> > > images > >> >> > > to > >> >> > > be > >> >> > > attached to whatever the caller of the query needs? > >> >> > > > >> >> > > - Ricky > >> >> > = > >> >> > i'm not sure they should be added automatically. > >> >> > I think a dialog[1] showing orphan disks/images on the > >> >> > storage > >> >> > domain > >> >> > for user to choose which to import as 'floating' disks would > >> >> > be > >> >> > better > >> >> > than auto importing them. > >> >> > = > >> >> > there is also the reverse of flagging existing disks as > >> >> > 'missing' > >> >> > in > >> >> > storage? > >> >> > = > >> >> = > >> >> Perhaps we should start a feature page to discuss and better > >> >> scope > >> >> it. > >> >> There is a feature page that we could expand, it doesn't > >> >> discuss > >> >> the > >> >> notion of importing those disks which is certainly something we > >> >> need > >> >> to address. > >> >> = > >> >> = > >> >> http://wiki.ovirt.org/wiki/Features/Orphaned_Images > >> > > >> >The original idea was to scan the storage domains and compare the > >> >images > >> >lists to the database, thus getting a list of images no longer > >> >relevant > >> >and scrubbing the storage. This will actually be addressed > >> >properly > >> >in > >> >the future (Ayal can elaborate on that) but for now this is > >> >needed > >> >at > >> >least for that use case. > >> > > >> > > >> >As I understand, the conversation here is about trying to take an > >> >already > >> >populated SD (from another setup I suppose), scanning it and > >> >putting > >> >it > >> >into RHEV? > >> = > >> As I understood it, the purpose of this functionality wasn't to > >> find > >> images which should be removed from storage, but to find images on > >> the > >> domain that oVirt was unaware of and importing them for use (for > >> instance, > >> if a disk was created outside of oVirt on the domain). If one of > >> the > >> use > >> cases for this feature is also the orphaned images mentioned on > >> the > >> feature page, that may expand the functionality into a separate > >> domain > >> scrub and storage import, both of which would call the rescan > >> (meaning the > >> rescan would not actually add to the database, but instead return > >> a > >> list > >> of "orphaned" disk images). > >> = > >> Another solution would be to import all disk images into the > >> database > >> either way, and let the user delete any orphaned images from the > >> GUI. > > > >I think are nice to have, but the problem with the scanning is that > >if > >we're not scanning a master domain or an export domain, all we will > >see > >is a bunch of images with no context or even hints as to where they > >belong. The data that makes it all usable is in the engine database > >and > >in the ovf files on the master domain. > > > >This is why I stopped at the orphaned images part of the feature - > >because there it's feasible, I would rely on the engine database for > >image ID comparisons. > > > >If we present a user with a list of nameless disks, I doubt it will > >be of > >any use. > = > The way this would work is by comparing a list of disk images from > vdsm > and from oVirt's database, finding the ones vdsm returns that oVirt > doesn't have, and then either adding or returning those images. So > oVirt's > db will be used in the comparison. This will work only when scanning storage domains already attached and in u= se by the current oVirt setup. What I am talking about is what will happen = if a LUN that used to be a SD in another oVirt setup is discovered and scan= ned, with no engine db to compare with. If we don't consider such a use cas= e, life is definitely quite easy, and we're basically within the scope of t= he orphaned images feature > = > As far as presenting the user with nameless disks, that's a point I > hadn't > considered; we could generate some sort of placeholder metadata upon > addition to show the user that these are new/orphaned disks that were > found on the storage domain. Is it safe to assume that the disks > discovered by this feature won't be attached to anything? The oVirt paradigm says "if it isn't in the engine db, it's not ours", so a= ny LV or image we discover that is missing from the DB or the snapshot chai= n of the image in the DB, is nameless, and orphaned. Such an image on a current SD, belonging to a working oVirt setup is defini= tely an orphaned image. Attaching these to VMs is usually also useless, bec= ause they are more often than not discarded snapshots that didn't get disca= rded cleanly for some reason. = Now, if we want to make this usable, we might want to actually check the qc= ow2 metadata of the image to see whether it's a mid-chain snapshot (and if = so it's probably just a candidate for cleanup), or a standalone qcow2 or ra= w image, and then we can move on with the virt-* tools, to find out the ima= ge size and the filesystems it contains. This will at least provide the use= r with some usable information about the detected image. If we're talking about scanning an SD that doesn't presently belong to the = current oVirt setup, then this is even more relevant, because all of the im= ages will have no VM-related context. > > = > > > >> > > >> >> = > >> >> > = > >> >> > [1] or a subtab on the storage domain. > >> >> > = > >> >> > _______________________________________________ > >> >> > Engine-devel mailing list > >> >> > Engine-devel(a)ovirt.org > >> >> > http://lists.ovirt.org/mailman/listinfo/engine-devel > >> >> > = > >> >> = > >> > > >> >-- > >> > > >> > > >> > > >> >Regards, > >> > > >> >Dan Yasny > >> >Red Hat Israel > >> >+972 9769 2280 > >> = > >> = > > > >-- > > > > > > > >Regards, > > > >Dan Yasny > >Red Hat Israel > >+972 9769 2280 > = > = -- = Regards, = Dan Yasny = Red Hat Israel = +972 9769 2280 --===============4411213061235486299==-- From Ricky.Hopper at netapp.com Wed Aug 1 13:36:27 2012 Content-Type: multipart/mixed; boundary="===============2873869656525979209==" MIME-Version: 1.0 From: Hopper, Ricky To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Wed, 01 Aug 2012 17:36:09 +0000 Message-ID: In-Reply-To: 1235552045.4962754.1343829915979.JavaMail.root@redhat.com --===============2873869656525979209== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 8/1/12 10:05 AM, "Dan Yasny" wrote: > > >----- Original Message ----- >> From: "Ricky Hopper" >> To: "Dan Yasny" >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , "Andrew >>Cathrow" >> Sent: Wednesday, 1 August, 2012 4:56:45 PM >> Subject: Re: [Engine-devel] Domain rescan action question >> = >> = >> = >> On 8/1/12 9:42 AM, "Dan Yasny" wrote: >> = >> > >> > >> >----- Original Message ----- >> >> From: "Ricky Hopper" >> >> To: "Dan Yasny" , "Andrew Cathrow" >> >> >> >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , >> >> "Ricky >> >>Hopper" >> >> Sent: Wednesday, 1 August, 2012 4:34:53 PM >> >> Subject: Re: [Engine-devel] Domain rescan action question >> >> = >> >> = >> >> = >> >> On 8/1/12 5:59 AM, "Dan Yasny" wrote: >> >> = >> >> > >> >> > >> >> >----- Original Message ----- >> >> >> From: "Andrew Cathrow" >> >> >> To: "Itamar Heim" , "Dan Yasny" >> >> >> , >> >> >>"Ricky Hopper" >> >> >> Cc: engine-devel(a)ovirt.org >> >> >> Sent: Wednesday, 1 August, 2012 12:24:42 AM >> >> >> Subject: Re: [Engine-devel] Domain rescan action question >> >> >> = >> >> >> = >> >> >> = >> >> >> ----- Original Message ----- >> >> >> > From: "Itamar Heim" >> >> >> > To: "Ricky Hopper" >> >> >> > Cc: engine-devel(a)ovirt.org >> >> >> > Sent: Tuesday, July 31, 2012 4:44:34 PM >> >> >> > Subject: Re: [Engine-devel] Domain rescan action question >> >> >> > = >> >> >> > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: >> >> >> > > Hey all, >> >> >> > > >> >> >> > > As I'm making progress with the domain rescan >> >> >> > > functionality, >> >> >> > > I've >> >> >> > > realized that I'm unsure what to do with any disks that are >> >> >> > > detected on >> >> >> > > the domain. Should I add them back into the database to be >> >> >> > > listed >> >> >> > > as >> >> >> > > floating disks, or should I just return a list of disk >> >> >> > > images >> >> >> > > to >> >> >> > > be >> >> >> > > attached to whatever the caller of the query needs? >> >> >> > > >> >> >> > > - Ricky >> >> >> > = >> >> >> > i'm not sure they should be added automatically. >> >> >> > I think a dialog[1] showing orphan disks/images on the >> >> >> > storage >> >> >> > domain >> >> >> > for user to choose which to import as 'floating' disks would >> >> >> > be >> >> >> > better >> >> >> > than auto importing them. >> >> >> > = >> >> >> > there is also the reverse of flagging existing disks as >> >> >> > 'missing' >> >> >> > in >> >> >> > storage? >> >> >> > = >> >> >> = >> >> >> Perhaps we should start a feature page to discuss and better >> >> >> scope >> >> >> it. >> >> >> There is a feature page that we could expand, it doesn't >> >> >> discuss >> >> >> the >> >> >> notion of importing those disks which is certainly something we >> >> >> need >> >> >> to address. >> >> >> = >> >> >> = >> >> >> http://wiki.ovirt.org/wiki/Features/Orphaned_Images >> >> > >> >> >The original idea was to scan the storage domains and compare the >> >> >images >> >> >lists to the database, thus getting a list of images no longer >> >> >relevant >> >> >and scrubbing the storage. This will actually be addressed >> >> >properly >> >> >in >> >> >the future (Ayal can elaborate on that) but for now this is >> >> >needed >> >> >at >> >> >least for that use case. >> >> > >> >> > >> >> >As I understand, the conversation here is about trying to take an >> >> >already >> >> >populated SD (from another setup I suppose), scanning it and >> >> >putting >> >> >it >> >> >into RHEV? >> >> = >> >> As I understood it, the purpose of this functionality wasn't to >> >> find >> >> images which should be removed from storage, but to find images on >> >> the >> >> domain that oVirt was unaware of and importing them for use (for >> >> instance, >> >> if a disk was created outside of oVirt on the domain). If one of >> >> the >> >> use >> >> cases for this feature is also the orphaned images mentioned on >> >> the >> >> feature page, that may expand the functionality into a separate >> >> domain >> >> scrub and storage import, both of which would call the rescan >> >> (meaning the >> >> rescan would not actually add to the database, but instead return >> >> a >> >> list >> >> of "orphaned" disk images). >> >> = >> >> Another solution would be to import all disk images into the >> >> database >> >> either way, and let the user delete any orphaned images from the >> >> GUI. >> > >> >I think are nice to have, but the problem with the scanning is that >> >if >> >we're not scanning a master domain or an export domain, all we will >> >see >> >is a bunch of images with no context or even hints as to where they >> >belong. The data that makes it all usable is in the engine database >> >and >> >in the ovf files on the master domain. >> > >> >This is why I stopped at the orphaned images part of the feature - >> >because there it's feasible, I would rely on the engine database for >> >image ID comparisons. >> > >> >If we present a user with a list of nameless disks, I doubt it will >> >be of >> >any use. >> = >> The way this would work is by comparing a list of disk images from >> vdsm >> and from oVirt's database, finding the ones vdsm returns that oVirt >> doesn't have, and then either adding or returning those images. So >> oVirt's >> db will be used in the comparison. > >This will work only when scanning storage domains already attached and in >use by the current oVirt setup. What I am talking about is what will >happen if a LUN that used to be a SD in another oVirt setup is discovered >and scanned, with no engine db to compare with. If we don't consider such >a use case, life is definitely quite easy, and we're basically within the >scope of the orphaned images feature This use case should definitely be considered, maybe have a separate case where the rescan would return all "compatible" disks (i.e. disks that aren't just partial snapshots and the like) if the domain has not yet been mounted. Essentially, it would run the same comparison, but compare against an empty list rather than a list of disks. There's no way it's as simple as that (I'm unsure of the methods oVirt uses to mount a domain), but it's a good starting point. > >> = >> As far as presenting the user with nameless disks, that's a point I >> hadn't >> considered; we could generate some sort of placeholder metadata upon >> addition to show the user that these are new/orphaned disks that were >> found on the storage domain. Is it safe to assume that the disks >> discovered by this feature won't be attached to anything? > >The oVirt paradigm says "if it isn't in the engine db, it's not ours", so >any LV or image we discover that is missing from the DB or the snapshot >chain of the image in the DB, is nameless, and orphaned. > >Such an image on a current SD, belonging to a working oVirt setup is >definitely an orphaned image. Attaching these to VMs is usually also >useless, because they are more often than not discarded snapshots that >didn't get discarded cleanly for some reason. > > >Now, if we want to make this usable, we might want to actually check the >qcow2 metadata of the image to see whether it's a mid-chain snapshot (and >if so it's probably just a candidate for cleanup), or a standalone qcow2 >or raw image, and then we can move on with the virt-* tools, to find out >the image size and the filesystems it contains. This will at least >provide the user with some usable information about the detected image. >If we're talking about scanning an SD that doesn't presently belong to >the current oVirt setup, then this is even more relevant, because all of >the images will have no VM-related context. We're currently working on having disks created outside of the oVirt environment, so not all orphaned disks on the existing storage domain will be artifacts of supposedly-deleted data. For our use case, disk images created by us will be able to be imported into oVirt and attached to a VM created through the engine. Because of this, saying "if it isn't in the engine db, it's not ours" wouldn't necessarily be true. When you talk about checking the metadata, does either oVirt or vdsm have a simple way to do this? A query of some sort would be ideal for this, as it could be run for each image as a qualifier for import. Also, as far as writing the functionality itself, I'm gathering that it should be structured as a query to return these orphaned images, which can then be acted upon/added to the database through a separate command after checking the validity of each image? > >> > = >> > >> >> > >> >> >> = >> >> >> > = >> >> >> > [1] or a subtab on the storage domain. >> >> >> > = >> >> >> > _______________________________________________ >> >> >> > Engine-devel mailing list >> >> >> > Engine-devel(a)ovirt.org >> >> >> > http://lists.ovirt.org/mailman/listinfo/engine-devel >> >> >> > = >> >> >> = >> >> > >> >> >-- >> >> > >> >> > >> >> > >> >> >Regards, >> >> > >> >> >Dan Yasny >> >> >Red Hat Israel >> >> >+972 9769 2280 >> >> = >> >> = >> > >> >-- >> > >> > >> > >> >Regards, >> > >> >Dan Yasny >> >Red Hat Israel >> >+972 9769 2280 >> = >> = > >-- = > > > >Regards, = > >Dan Yasny = >Red Hat Israel = >+972 9769 2280 --===============2873869656525979209==-- From dyasny at redhat.com Wed Aug 1 14:35:31 2012 Content-Type: multipart/mixed; boundary="===============0360332290351666251==" MIME-Version: 1.0 From: Dan Yasny To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Wed, 01 Aug 2012 14:35:29 -0400 Message-ID: <1952269421.5171062.1343846129879.JavaMail.root@redhat.com> In-Reply-To: CC3EB4BC.14A9%rickyh@netapp.com --===============0360332290351666251== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----- Original Message ----- > From: "Ricky Hopper" > To: "Dan Yasny" > Cc: engine-devel(a)ovirt.org, "Itamar Heim" , "Andrew= Cathrow" > Sent: Wednesday, 1 August, 2012 8:36:09 PM > Subject: Re: [Engine-devel] Domain rescan action question > = > = > = > On 8/1/12 10:05 AM, "Dan Yasny" wrote: > = > > > > > >----- Original Message ----- > >> From: "Ricky Hopper" > >> To: "Dan Yasny" > >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , > >> "Andrew > >>Cathrow" > >> Sent: Wednesday, 1 August, 2012 4:56:45 PM > >> Subject: Re: [Engine-devel] Domain rescan action question > >> = > >> = > >> = > >> On 8/1/12 9:42 AM, "Dan Yasny" wrote: > >> = > >> > > >> > > >> >----- Original Message ----- > >> >> From: "Ricky Hopper" > >> >> To: "Dan Yasny" , "Andrew Cathrow" > >> >> > >> >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , > >> >> "Ricky > >> >>Hopper" > >> >> Sent: Wednesday, 1 August, 2012 4:34:53 PM > >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> = > >> >> = > >> >> = > >> >> On 8/1/12 5:59 AM, "Dan Yasny" wrote: > >> >> = > >> >> > > >> >> > > >> >> >----- Original Message ----- > >> >> >> From: "Andrew Cathrow" > >> >> >> To: "Itamar Heim" , "Dan Yasny" > >> >> >> , > >> >> >>"Ricky Hopper" > >> >> >> Cc: engine-devel(a)ovirt.org > >> >> >> Sent: Wednesday, 1 August, 2012 12:24:42 AM > >> >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> >> = > >> >> >> = > >> >> >> = > >> >> >> ----- Original Message ----- > >> >> >> > From: "Itamar Heim" > >> >> >> > To: "Ricky Hopper" > >> >> >> > Cc: engine-devel(a)ovirt.org > >> >> >> > Sent: Tuesday, July 31, 2012 4:44:34 PM > >> >> >> > Subject: Re: [Engine-devel] Domain rescan action question > >> >> >> > = > >> >> >> > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > >> >> >> > > Hey all, > >> >> >> > > > >> >> >> > > As I'm making progress with the domain rescan > >> >> >> > > functionality, > >> >> >> > > I've > >> >> >> > > realized that I'm unsure what to do with any disks that > >> >> >> > > are > >> >> >> > > detected on > >> >> >> > > the domain. Should I add them back into the database to > >> >> >> > > be > >> >> >> > > listed > >> >> >> > > as > >> >> >> > > floating disks, or should I just return a list of disk > >> >> >> > > images > >> >> >> > > to > >> >> >> > > be > >> >> >> > > attached to whatever the caller of the query needs? > >> >> >> > > > >> >> >> > > - Ricky > >> >> >> > = > >> >> >> > i'm not sure they should be added automatically. > >> >> >> > I think a dialog[1] showing orphan disks/images on the > >> >> >> > storage > >> >> >> > domain > >> >> >> > for user to choose which to import as 'floating' disks > >> >> >> > would > >> >> >> > be > >> >> >> > better > >> >> >> > than auto importing them. > >> >> >> > = > >> >> >> > there is also the reverse of flagging existing disks as > >> >> >> > 'missing' > >> >> >> > in > >> >> >> > storage? > >> >> >> > = > >> >> >> = > >> >> >> Perhaps we should start a feature page to discuss and better > >> >> >> scope > >> >> >> it. > >> >> >> There is a feature page that we could expand, it doesn't > >> >> >> discuss > >> >> >> the > >> >> >> notion of importing those disks which is certainly something > >> >> >> we > >> >> >> need > >> >> >> to address. > >> >> >> = > >> >> >> = > >> >> >> http://wiki.ovirt.org/wiki/Features/Orphaned_Images > >> >> > > >> >> >The original idea was to scan the storage domains and compare > >> >> >the > >> >> >images > >> >> >lists to the database, thus getting a list of images no longer > >> >> >relevant > >> >> >and scrubbing the storage. This will actually be addressed > >> >> >properly > >> >> >in > >> >> >the future (Ayal can elaborate on that) but for now this is > >> >> >needed > >> >> >at > >> >> >least for that use case. > >> >> > > >> >> > > >> >> >As I understand, the conversation here is about trying to take > >> >> >an > >> >> >already > >> >> >populated SD (from another setup I suppose), scanning it and > >> >> >putting > >> >> >it > >> >> >into RHEV? > >> >> = > >> >> As I understood it, the purpose of this functionality wasn't to > >> >> find > >> >> images which should be removed from storage, but to find images > >> >> on > >> >> the > >> >> domain that oVirt was unaware of and importing them for use > >> >> (for > >> >> instance, > >> >> if a disk was created outside of oVirt on the domain). If one > >> >> of > >> >> the > >> >> use > >> >> cases for this feature is also the orphaned images mentioned on > >> >> the > >> >> feature page, that may expand the functionality into a separate > >> >> domain > >> >> scrub and storage import, both of which would call the rescan > >> >> (meaning the > >> >> rescan would not actually add to the database, but instead > >> >> return > >> >> a > >> >> list > >> >> of "orphaned" disk images). > >> >> = > >> >> Another solution would be to import all disk images into the > >> >> database > >> >> either way, and let the user delete any orphaned images from > >> >> the > >> >> GUI. > >> > > >> >I think are nice to have, but the problem with the scanning is > >> >that > >> >if > >> >we're not scanning a master domain or an export domain, all we > >> >will > >> >see > >> >is a bunch of images with no context or even hints as to where > >> >they > >> >belong. The data that makes it all usable is in the engine > >> >database > >> >and > >> >in the ovf files on the master domain. > >> > > >> >This is why I stopped at the orphaned images part of the feature > >> >- > >> >because there it's feasible, I would rely on the engine database > >> >for > >> >image ID comparisons. > >> > > >> >If we present a user with a list of nameless disks, I doubt it > >> >will > >> >be of > >> >any use. > >> = > >> The way this would work is by comparing a list of disk images from > >> vdsm > >> and from oVirt's database, finding the ones vdsm returns that > >> oVirt > >> doesn't have, and then either adding or returning those images. So > >> oVirt's > >> db will be used in the comparison. > > > >This will work only when scanning storage domains already attached > >and in > >use by the current oVirt setup. What I am talking about is what will > >happen if a LUN that used to be a SD in another oVirt setup is > >discovered > >and scanned, with no engine db to compare with. If we don't consider > >such > >a use case, life is definitely quite easy, and we're basically > >within the > >scope of the orphaned images feature > = > This use case should definitely be considered, maybe have a separate > case > where the rescan would return all "compatible" disks (i.e. disks that > aren't just partial snapshots and the like) if the domain has not yet > been > mounted. Essentially, it would run the same comparison, but compare > against an empty list rather than a list of disks. There's no way > it's as > simple as that (I'm unsure of the methods oVirt uses to mount a > domain), > but it's a good starting point. There is no complex method there. For file storage it's just a mount comman= d, and for block it's LVM (plus iscsi session establishment, if needed) > > > >> = > >> As far as presenting the user with nameless disks, that's a point > >> I > >> hadn't > >> considered; we could generate some sort of placeholder metadata > >> upon > >> addition to show the user that these are new/orphaned disks that > >> were > >> found on the storage domain. Is it safe to assume that the disks > >> discovered by this feature won't be attached to anything? > > > >The oVirt paradigm says "if it isn't in the engine db, it's not > >ours", so > >any LV or image we discover that is missing from the DB or the > >snapshot > >chain of the image in the DB, is nameless, and orphaned. > > > >Such an image on a current SD, belonging to a working oVirt setup is > >definitely an orphaned image. Attaching these to VMs is usually also > >useless, because they are more often than not discarded snapshots > >that > >didn't get discarded cleanly for some reason. > > > > > >Now, if we want to make this usable, we might want to actually check > >the > >qcow2 metadata of the image to see whether it's a mid-chain snapshot > >(and > >if so it's probably just a candidate for cleanup), or a standalone > >qcow2 > >or raw image, and then we can move on with the virt-* tools, to find > >out > >the image size and the filesystems it contains. This will at least > >provide the user with some usable information about the detected > >image. > >If we're talking about scanning an SD that doesn't presently belong > >to > >the current oVirt setup, then this is even more relevant, because > >all of > >the images will have no VM-related context. > = > We're currently working on having disks created outside of the oVirt > environment, so not all orphaned disks on the existing storage domain > will > be artifacts of supposedly-deleted data. = Do you mean like rhevm-image-upload, or something different? > For our use case, disk > images > created by us will be able to be imported into oVirt and attached to > a VM > created through the engine. Because of this, saying "if it isn't in > the > engine db, it's not ours" wouldn't necessarily be true. > = > When you talk about checking the metadata, does either oVirt or vdsm > have > a simple way to do this? A query of some sort would be ideal for > this, as > it could be run for each image as a qualifier for import. qemu-img info and libguestfs commands should do. Besides, our images do come with some metadata (in the LVM tags or a .meta = file) > = > Also, as far as writing the functionality itself, I'm gathering that > it > should be structured as a query to return these orphaned images, > which can > then be acted upon/added to the database through a separate command > after > checking the validity of each image? Yes, a simple way to say "import this one to DB, attach to VM X or make flo= ating", "delete that one", "skip" > > > >> > = > >> > > >> >> > > >> >> >> = > >> >> >> > = > >> >> >> > [1] or a subtab on the storage domain. > >> >> >> > = > >> >> >> > _______________________________________________ > >> >> >> > Engine-devel mailing list > >> >> >> > Engine-devel(a)ovirt.org > >> >> >> > http://lists.ovirt.org/mailman/listinfo/engine-devel > >> >> >> > = > >> >> >> = > >> >> > > >> >> >-- > >> >> > > >> >> > > >> >> > > >> >> >Regards, > >> >> > > >> >> >Dan Yasny > >> >> >Red Hat Israel > >> >> >+972 9769 2280 > >> >> = > >> >> = > >> > > >> >-- > >> > > >> > > >> > > >> >Regards, > >> > > >> >Dan Yasny > >> >Red Hat Israel > >> >+972 9769 2280 > >> = > >> = > > > >-- > > > > > > > >Regards, > > > >Dan Yasny > >Red Hat Israel > >+972 9769 2280 > = > = -- = Regards, = Dan Yasny = Red Hat Israel = +972 9769 2280 --===============0360332290351666251==-- From shuming at linux.vnet.ibm.com Wed Aug 1 21:45:39 2012 Content-Type: multipart/mixed; boundary="===============7926838817222657780==" MIME-Version: 1.0 From: Shu Ming To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Thu, 02 Aug 2012 09:44:56 +0800 Message-ID: <5019DB98.6050702@linux.vnet.ibm.com> In-Reply-To: CC3EB4BC.14A9%rickyh@netapp.com --===============7926838817222657780== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable No comments below. I am just curious how you can have disk created = outside of the oVirt environment if the storage domain is mounted on = oVirt engine. Does your application try to communicate with VDSM though = SPM host to do that? Are the two applications(oVirt engine and your = application) coexisting and manipulating the storage domain at the same = time or one application is quiet when the other one is functioning? = IMHO, modifying the storage domain meta data from two application are = dangerous and I suppose your application should know the reason and what = these disks are for. So telling oVirt to import the disks = automatically should be reasonable because you know what these disks are = for and it is unnecessary for the oVirt administrator to select which = one to import. On 2012-8-2 1:36, Hopper, Ricky wrote: > > On 8/1/12 10:05 AM, "Dan Yasny" wrote: > >> >> ----- Original Message ----- >>> From: "Ricky Hopper" >>> To: "Dan Yasny" >>> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , "Andr= ew >>> Cathrow" >>> Sent: Wednesday, 1 August, 2012 4:56:45 PM >>> Subject: Re: [Engine-devel] Domain rescan action question >>> >>> >>> >>> On 8/1/12 9:42 AM, "Dan Yasny" wrote: >>> >>>> >>>> ----- Original Message ----- >>>>> From: "Ricky Hopper" >>>>> To: "Dan Yasny" , "Andrew Cathrow" >>>>> >>>>> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , >>>>> "Ricky >>>>> Hopper" >>>>> Sent: Wednesday, 1 August, 2012 4:34:53 PM >>>>> Subject: Re: [Engine-devel] Domain rescan action question >>>>> >>>>> >>>>> >>>>> On 8/1/12 5:59 AM, "Dan Yasny" wrote: >>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Andrew Cathrow" >>>>>>> To: "Itamar Heim" , "Dan Yasny" >>>>>>> , >>>>>>> "Ricky Hopper" >>>>>>> Cc: engine-devel(a)ovirt.org >>>>>>> Sent: Wednesday, 1 August, 2012 12:24:42 AM >>>>>>> Subject: Re: [Engine-devel] Domain rescan action question >>>>>>> >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Itamar Heim" >>>>>>>> To: "Ricky Hopper" >>>>>>>> Cc: engine-devel(a)ovirt.org >>>>>>>> Sent: Tuesday, July 31, 2012 4:44:34 PM >>>>>>>> Subject: Re: [Engine-devel] Domain rescan action question >>>>>>>> >>>>>>>> On 07/31/2012 11:30 PM, Hopper, Ricky wrote: >>>>>>>>> Hey all, >>>>>>>>> >>>>>>>>> As I'm making progress with the domain rescan >>>>>>>>> functionality, >>>>>>>>> I've >>>>>>>>> realized that I'm unsure what to do with any disks that are >>>>>>>>> detected on >>>>>>>>> the domain. Should I add them back into the database to be >>>>>>>>> listed >>>>>>>>> as >>>>>>>>> floating disks, or should I just return a list of disk >>>>>>>>> images >>>>>>>>> to >>>>>>>>> be >>>>>>>>> attached to whatever the caller of the query needs? >>>>>>>>> >>>>>>>>> - Ricky >>>>>>>> i'm not sure they should be added automatically. >>>>>>>> I think a dialog[1] showing orphan disks/images on the >>>>>>>> storage >>>>>>>> domain >>>>>>>> for user to choose which to import as 'floating' disks would >>>>>>>> be >>>>>>>> better >>>>>>>> than auto importing them. >>>>>>>> >>>>>>>> there is also the reverse of flagging existing disks as >>>>>>>> 'missing' >>>>>>>> in >>>>>>>> storage? >>>>>>>> >>>>>>> Perhaps we should start a feature page to discuss and better >>>>>>> scope >>>>>>> it. >>>>>>> There is a feature page that we could expand, it doesn't >>>>>>> discuss >>>>>>> the >>>>>>> notion of importing those disks which is certainly something we >>>>>>> need >>>>>>> to address. >>>>>>> >>>>>>> >>>>>>> http://wiki.ovirt.org/wiki/Features/Orphaned_Images >>>>>> The original idea was to scan the storage domains and compare the >>>>>> images >>>>>> lists to the database, thus getting a list of images no longer >>>>>> relevant >>>>>> and scrubbing the storage. This will actually be addressed >>>>>> properly >>>>>> in >>>>>> the future (Ayal can elaborate on that) but for now this is >>>>>> needed >>>>>> at >>>>>> least for that use case. >>>>>> >>>>>> >>>>>> As I understand, the conversation here is about trying to take an >>>>>> already >>>>>> populated SD (from another setup I suppose), scanning it and >>>>>> putting >>>>>> it >>>>>> into RHEV? >>>>> As I understood it, the purpose of this functionality wasn't to >>>>> find >>>>> images which should be removed from storage, but to find images on >>>>> the >>>>> domain that oVirt was unaware of and importing them for use (for >>>>> instance, >>>>> if a disk was created outside of oVirt on the domain). If one of >>>>> the >>>>> use >>>>> cases for this feature is also the orphaned images mentioned on >>>>> the >>>>> feature page, that may expand the functionality into a separate >>>>> domain >>>>> scrub and storage import, both of which would call the rescan >>>>> (meaning the >>>>> rescan would not actually add to the database, but instead return >>>>> a >>>>> list >>>>> of "orphaned" disk images). >>>>> >>>>> Another solution would be to import all disk images into the >>>>> database >>>>> either way, and let the user delete any orphaned images from the >>>>> GUI. >>>> I think are nice to have, but the problem with the scanning is that >>>> if >>>> we're not scanning a master domain or an export domain, all we will >>>> see >>>> is a bunch of images with no context or even hints as to where they >>>> belong. The data that makes it all usable is in the engine database >>>> and >>>> in the ovf files on the master domain. >>>> >>>> This is why I stopped at the orphaned images part of the feature - >>>> because there it's feasible, I would rely on the engine database for >>>> image ID comparisons. >>>> >>>> If we present a user with a list of nameless disks, I doubt it will >>>> be of >>>> any use. >>> The way this would work is by comparing a list of disk images from >>> vdsm >>> and from oVirt's database, finding the ones vdsm returns that oVirt >>> doesn't have, and then either adding or returning those images. So >>> oVirt's >>> db will be used in the comparison. >> This will work only when scanning storage domains already attached and in >> use by the current oVirt setup. What I am talking about is what will >> happen if a LUN that used to be a SD in another oVirt setup is discovered >> and scanned, with no engine db to compare with. If we don't consider such >> a use case, life is definitely quite easy, and we're basically within the >> scope of the orphaned images feature > This use case should definitely be considered, maybe have a separate case > where the rescan would return all "compatible" disks (i.e. disks that > aren't just partial snapshots and the like) if the domain has not yet been > mounted. Essentially, it would run the same comparison, but compare > against an empty list rather than a list of disks. There's no way it's as > simple as that (I'm unsure of the methods oVirt uses to mount a domain), > but it's a good starting point. >>> As far as presenting the user with nameless disks, that's a point I >>> hadn't >>> considered; we could generate some sort of placeholder metadata upon >>> addition to show the user that these are new/orphaned disks that were >>> found on the storage domain. Is it safe to assume that the disks >>> discovered by this feature won't be attached to anything? >> The oVirt paradigm says "if it isn't in the engine db, it's not ours", so >> any LV or image we discover that is missing from the DB or the snapshot >> chain of the image in the DB, is nameless, and orphaned. >> >> Such an image on a current SD, belonging to a working oVirt setup is >> definitely an orphaned image. Attaching these to VMs is usually also >> useless, because they are more often than not discarded snapshots that >> didn't get discarded cleanly for some reason. >> >> >> Now, if we want to make this usable, we might want to actually check the >> qcow2 metadata of the image to see whether it's a mid-chain snapshot (and >> if so it's probably just a candidate for cleanup), or a standalone qcow2 >> or raw image, and then we can move on with the virt-* tools, to find out >> the image size and the filesystems it contains. This will at least >> provide the user with some usable information about the detected image. >> If we're talking about scanning an SD that doesn't presently belong to >> the current oVirt setup, then this is even more relevant, because all of >> the images will have no VM-related context. > We're currently working on having disks created outside of the oVirt > environment, so not all orphaned disks on the existing storage domain will > be artifacts of supposedly-deleted data. For our use case, disk images > created by us will be able to be imported into oVirt and attached to a VM > created through the engine. Because of this, saying "if it isn't in the > engine db, it's not ours" wouldn't necessarily be true. > > When you talk about checking the metadata, does either oVirt or vdsm have > a simple way to do this? A query of some sort would be ideal for this, as > it could be run for each image as a qualifier for import. > > Also, as far as writing the functionality itself, I'm gathering that it > should be structured as a query to return these orphaned images, which can > then be acted upon/added to the database through a separate command after > checking the validity of each image? >>>> >>>>>>>> [1] or a subtab on the storage domain. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Engine-devel mailing list >>>>>>>> Engine-devel(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>>>>>>> >>>>>> -- >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Dan Yasny >>>>>> Red Hat Israel >>>>>> +972 9769 2280 >>>>> >>>> -- >>>> >>>> >>>> >>>> Regards, >>>> >>>> Dan Yasny >>>> Red Hat Israel >>>> +972 9769 2280 >>> >> -- = >> >> >> >> Regards, >> >> Dan Yasny >> Red Hat Israel >> +972 9769 2280 > _______________________________________________ > Engine-devel mailing list > Engine-devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > -- = Shu Ming IBM China Systems and Technology Laboratory --===============7926838817222657780==-- From iheim at redhat.com Thu Aug 2 02:59:42 2012 Content-Type: multipart/mixed; boundary="===============4003710898076851857==" MIME-Version: 1.0 From: Itamar Heim To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Thu, 02 Aug 2012 09:59:36 +0300 Message-ID: <501A2558.6000401@redhat.com> In-Reply-To: 5019DB98.6050702@linux.vnet.ibm.com --===============4003710898076851857== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 08/02/2012 04:44 AM, Shu Ming wrote: > No comments below. I am just curious how you can have disk created > outside of the oVirt environment if the storage domain is mounted on > oVirt engine. Does your application try to communicate with VDSM though > SPM host to do that? Are the two applications(oVirt engine and your > application) coexisting and manipulating the storage domain at the same > time or one application is quiet when the other one is functioning? > IMHO, modifying the storage domain meta data from two application are > dangerous and I suppose your application should know the reason and what > these disks are for. So telling oVirt to import the disks > automatically should be reasonable because you know what these disks are > for and it is unnecessary for the oVirt administrator to select which > one to import. two use cases: 1. NFS storage allows this quite easily, and very relevant with storage = side cloning. 2. engine and storage go out of sync (recover either from backup) > > On 2012-8-2 1:36, Hopper, Ricky wrote: >> >> On 8/1/12 10:05 AM, "Dan Yasny" wrote: >> >>> >>> ----- Original Message ----- >>>> From: "Ricky Hopper" >>>> To: "Dan Yasny" >>>> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , "And= rew >>>> Cathrow" >>>> Sent: Wednesday, 1 August, 2012 4:56:45 PM >>>> Subject: Re: [Engine-devel] Domain rescan action question >>>> >>>> >>>> >>>> On 8/1/12 9:42 AM, "Dan Yasny" wrote: >>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Ricky Hopper" >>>>>> To: "Dan Yasny" , "Andrew Cathrow" >>>>>> >>>>>> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , >>>>>> "Ricky >>>>>> Hopper" >>>>>> Sent: Wednesday, 1 August, 2012 4:34:53 PM >>>>>> Subject: Re: [Engine-devel] Domain rescan action question >>>>>> >>>>>> >>>>>> >>>>>> On 8/1/12 5:59 AM, "Dan Yasny" wrote: >>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Andrew Cathrow" >>>>>>>> To: "Itamar Heim" , "Dan Yasny" >>>>>>>> , >>>>>>>> "Ricky Hopper" >>>>>>>> Cc: engine-devel(a)ovirt.org >>>>>>>> Sent: Wednesday, 1 August, 2012 12:24:42 AM >>>>>>>> Subject: Re: [Engine-devel] Domain rescan action question >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Itamar Heim" >>>>>>>>> To: "Ricky Hopper" >>>>>>>>> Cc: engine-devel(a)ovirt.org >>>>>>>>> Sent: Tuesday, July 31, 2012 4:44:34 PM >>>>>>>>> Subject: Re: [Engine-devel] Domain rescan action question >>>>>>>>> >>>>>>>>> On 07/31/2012 11:30 PM, Hopper, Ricky wrote: >>>>>>>>>> Hey all, >>>>>>>>>> >>>>>>>>>> As I'm making progress with the domain rescan >>>>>>>>>> functionality, >>>>>>>>>> I've >>>>>>>>>> realized that I'm unsure what to do with any disks that are >>>>>>>>>> detected on >>>>>>>>>> the domain. Should I add them back into the database to be >>>>>>>>>> listed >>>>>>>>>> as >>>>>>>>>> floating disks, or should I just return a list of disk >>>>>>>>>> images >>>>>>>>>> to >>>>>>>>>> be >>>>>>>>>> attached to whatever the caller of the query needs? >>>>>>>>>> >>>>>>>>>> - Ricky >>>>>>>>> i'm not sure they should be added automatically. >>>>>>>>> I think a dialog[1] showing orphan disks/images on the >>>>>>>>> storage >>>>>>>>> domain >>>>>>>>> for user to choose which to import as 'floating' disks would >>>>>>>>> be >>>>>>>>> better >>>>>>>>> than auto importing them. >>>>>>>>> >>>>>>>>> there is also the reverse of flagging existing disks as >>>>>>>>> 'missing' >>>>>>>>> in >>>>>>>>> storage? >>>>>>>>> >>>>>>>> Perhaps we should start a feature page to discuss and better >>>>>>>> scope >>>>>>>> it. >>>>>>>> There is a feature page that we could expand, it doesn't >>>>>>>> discuss >>>>>>>> the >>>>>>>> notion of importing those disks which is certainly something we >>>>>>>> need >>>>>>>> to address. >>>>>>>> >>>>>>>> >>>>>>>> http://wiki.ovirt.org/wiki/Features/Orphaned_Images >>>>>>> The original idea was to scan the storage domains and compare the >>>>>>> images >>>>>>> lists to the database, thus getting a list of images no longer >>>>>>> relevant >>>>>>> and scrubbing the storage. This will actually be addressed >>>>>>> properly >>>>>>> in >>>>>>> the future (Ayal can elaborate on that) but for now this is >>>>>>> needed >>>>>>> at >>>>>>> least for that use case. >>>>>>> >>>>>>> >>>>>>> As I understand, the conversation here is about trying to take an >>>>>>> already >>>>>>> populated SD (from another setup I suppose), scanning it and >>>>>>> putting >>>>>>> it >>>>>>> into RHEV? >>>>>> As I understood it, the purpose of this functionality wasn't to >>>>>> find >>>>>> images which should be removed from storage, but to find images on >>>>>> the >>>>>> domain that oVirt was unaware of and importing them for use (for >>>>>> instance, >>>>>> if a disk was created outside of oVirt on the domain). If one of >>>>>> the >>>>>> use >>>>>> cases for this feature is also the orphaned images mentioned on >>>>>> the >>>>>> feature page, that may expand the functionality into a separate >>>>>> domain >>>>>> scrub and storage import, both of which would call the rescan >>>>>> (meaning the >>>>>> rescan would not actually add to the database, but instead return >>>>>> a >>>>>> list >>>>>> of "orphaned" disk images). >>>>>> >>>>>> Another solution would be to import all disk images into the >>>>>> database >>>>>> either way, and let the user delete any orphaned images from the >>>>>> GUI. >>>>> I think are nice to have, but the problem with the scanning is that >>>>> if >>>>> we're not scanning a master domain or an export domain, all we will >>>>> see >>>>> is a bunch of images with no context or even hints as to where they >>>>> belong. The data that makes it all usable is in the engine database >>>>> and >>>>> in the ovf files on the master domain. >>>>> >>>>> This is why I stopped at the orphaned images part of the feature - >>>>> because there it's feasible, I would rely on the engine database for >>>>> image ID comparisons. >>>>> >>>>> If we present a user with a list of nameless disks, I doubt it will >>>>> be of >>>>> any use. >>>> The way this would work is by comparing a list of disk images from >>>> vdsm >>>> and from oVirt's database, finding the ones vdsm returns that oVirt >>>> doesn't have, and then either adding or returning those images. So >>>> oVirt's >>>> db will be used in the comparison. >>> This will work only when scanning storage domains already attached >>> and in >>> use by the current oVirt setup. What I am talking about is what will >>> happen if a LUN that used to be a SD in another oVirt setup is >>> discovered >>> and scanned, with no engine db to compare with. If we don't consider >>> such >>> a use case, life is definitely quite easy, and we're basically within >>> the >>> scope of the orphaned images feature >> This use case should definitely be considered, maybe have a separate case >> where the rescan would return all "compatible" disks (i.e. disks that >> aren't just partial snapshots and the like) if the domain has not yet >> been >> mounted. Essentially, it would run the same comparison, but compare >> against an empty list rather than a list of disks. There's no way it's as >> simple as that (I'm unsure of the methods oVirt uses to mount a domain), >> but it's a good starting point. >>>> As far as presenting the user with nameless disks, that's a point I >>>> hadn't >>>> considered; we could generate some sort of placeholder metadata upon >>>> addition to show the user that these are new/orphaned disks that were >>>> found on the storage domain. Is it safe to assume that the disks >>>> discovered by this feature won't be attached to anything? >>> The oVirt paradigm says "if it isn't in the engine db, it's not >>> ours", so >>> any LV or image we discover that is missing from the DB or the snapshot >>> chain of the image in the DB, is nameless, and orphaned. >>> >>> Such an image on a current SD, belonging to a working oVirt setup is >>> definitely an orphaned image. Attaching these to VMs is usually also >>> useless, because they are more often than not discarded snapshots that >>> didn't get discarded cleanly for some reason. >>> >>> >>> Now, if we want to make this usable, we might want to actually check the >>> qcow2 metadata of the image to see whether it's a mid-chain snapshot >>> (and >>> if so it's probably just a candidate for cleanup), or a standalone qcow2 >>> or raw image, and then we can move on with the virt-* tools, to find out >>> the image size and the filesystems it contains. This will at least >>> provide the user with some usable information about the detected image. >>> If we're talking about scanning an SD that doesn't presently belong to >>> the current oVirt setup, then this is even more relevant, because all of >>> the images will have no VM-related context. >> We're currently working on having disks created outside of the oVirt >> environment, so not all orphaned disks on the existing storage domain >> will >> be artifacts of supposedly-deleted data. For our use case, disk images >> created by us will be able to be imported into oVirt and attached to a VM >> created through the engine. Because of this, saying "if it isn't in the >> engine db, it's not ours" wouldn't necessarily be true. >> >> When you talk about checking the metadata, does either oVirt or vdsm have >> a simple way to do this? A query of some sort would be ideal for this, as >> it could be run for each image as a qualifier for import. >> >> Also, as far as writing the functionality itself, I'm gathering that it >> should be structured as a query to return these orphaned images, which >> can >> then be acted upon/added to the database through a separate command after >> checking the validity of each image? >>>>> >>>>>>>>> [1] or a subtab on the storage domain. >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Engine-devel mailing list >>>>>>>>> Engine-devel(a)ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>>>>>>>> >>>>>>> -- >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Dan Yasny >>>>>>> Red Hat Israel >>>>>>> +972 9769 2280 >>>>>> >>>>> -- >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Dan Yasny >>>>> Red Hat Israel >>>>> +972 9769 2280 >>>> >>> -- >>> >>> >>> >>> Regards, >>> >>> Dan Yasny >>> Red Hat Israel >>> +972 9769 2280 >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >> > > --===============4003710898076851857==-- From Ricky.Hopper at netapp.com Thu Aug 2 10:37:27 2012 Content-Type: multipart/mixed; boundary="===============8565674956477340264==" MIME-Version: 1.0 From: Hopper, Ricky To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Thu, 02 Aug 2012 14:37:10 +0000 Message-ID: In-Reply-To: 1952269421.5171062.1343846129879.JavaMail.root@redhat.com --===============8565674956477340264== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable In the interest of good discussion, we've put up a feature page for this feature (http://wiki.ovirt.org/wiki/Features/Domain_Scan), which links to a talk page where modifications can be proposed to how I've laid out the feature. So far, it covers how the query works and which commands will come about to implement it. I'd appreciate it if anyone concerned could check this out and make any changes as they see fit so we can get going with the coding. - Ricky On 8/1/12 2:35 PM, "Dan Yasny" wrote: > > >----- Original Message ----- >> From: "Ricky Hopper" >> To: "Dan Yasny" >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , "Andrew >>Cathrow" >> Sent: Wednesday, 1 August, 2012 8:36:09 PM >> Subject: Re: [Engine-devel] Domain rescan action question >> = >> = >> = >> On 8/1/12 10:05 AM, "Dan Yasny" wrote: >> = >> > >> > >> >----- Original Message ----- >> >> From: "Ricky Hopper" >> >> To: "Dan Yasny" >> >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , >> >> "Andrew >> >>Cathrow" >> >> Sent: Wednesday, 1 August, 2012 4:56:45 PM >> >> Subject: Re: [Engine-devel] Domain rescan action question >> >> = >> >> = >> >> = >> >> On 8/1/12 9:42 AM, "Dan Yasny" wrote: >> >> = >> >> > >> >> > >> >> >----- Original Message ----- >> >> >> From: "Ricky Hopper" >> >> >> To: "Dan Yasny" , "Andrew Cathrow" >> >> >> >> >> >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , >> >> >> "Ricky >> >> >>Hopper" >> >> >> Sent: Wednesday, 1 August, 2012 4:34:53 PM >> >> >> Subject: Re: [Engine-devel] Domain rescan action question >> >> >> = >> >> >> = >> >> >> = >> >> >> On 8/1/12 5:59 AM, "Dan Yasny" wrote: >> >> >> = >> >> >> > >> >> >> > >> >> >> >----- Original Message ----- >> >> >> >> From: "Andrew Cathrow" >> >> >> >> To: "Itamar Heim" , "Dan Yasny" >> >> >> >> , >> >> >> >>"Ricky Hopper" >> >> >> >> Cc: engine-devel(a)ovirt.org >> >> >> >> Sent: Wednesday, 1 August, 2012 12:24:42 AM >> >> >> >> Subject: Re: [Engine-devel] Domain rescan action question >> >> >> >> = >> >> >> >> = >> >> >> >> = >> >> >> >> ----- Original Message ----- >> >> >> >> > From: "Itamar Heim" >> >> >> >> > To: "Ricky Hopper" >> >> >> >> > Cc: engine-devel(a)ovirt.org >> >> >> >> > Sent: Tuesday, July 31, 2012 4:44:34 PM >> >> >> >> > Subject: Re: [Engine-devel] Domain rescan action question >> >> >> >> > = >> >> >> >> > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: >> >> >> >> > > Hey all, >> >> >> >> > > >> >> >> >> > > As I'm making progress with the domain rescan >> >> >> >> > > functionality, >> >> >> >> > > I've >> >> >> >> > > realized that I'm unsure what to do with any disks that >> >> >> >> > > are >> >> >> >> > > detected on >> >> >> >> > > the domain. Should I add them back into the database to >> >> >> >> > > be >> >> >> >> > > listed >> >> >> >> > > as >> >> >> >> > > floating disks, or should I just return a list of disk >> >> >> >> > > images >> >> >> >> > > to >> >> >> >> > > be >> >> >> >> > > attached to whatever the caller of the query needs? >> >> >> >> > > >> >> >> >> > > - Ricky >> >> >> >> > = >> >> >> >> > i'm not sure they should be added automatically. >> >> >> >> > I think a dialog[1] showing orphan disks/images on the >> >> >> >> > storage >> >> >> >> > domain >> >> >> >> > for user to choose which to import as 'floating' disks >> >> >> >> > would >> >> >> >> > be >> >> >> >> > better >> >> >> >> > than auto importing them. >> >> >> >> > = >> >> >> >> > there is also the reverse of flagging existing disks as >> >> >> >> > 'missing' >> >> >> >> > in >> >> >> >> > storage? >> >> >> >> > = >> >> >> >> = >> >> >> >> Perhaps we should start a feature page to discuss and better >> >> >> >> scope >> >> >> >> it. >> >> >> >> There is a feature page that we could expand, it doesn't >> >> >> >> discuss >> >> >> >> the >> >> >> >> notion of importing those disks which is certainly something >> >> >> >> we >> >> >> >> need >> >> >> >> to address. >> >> >> >> = >> >> >> >> = >> >> >> >> http://wiki.ovirt.org/wiki/Features/Orphaned_Images >> >> >> > >> >> >> >The original idea was to scan the storage domains and compare >> >> >> >the >> >> >> >images >> >> >> >lists to the database, thus getting a list of images no longer >> >> >> >relevant >> >> >> >and scrubbing the storage. This will actually be addressed >> >> >> >properly >> >> >> >in >> >> >> >the future (Ayal can elaborate on that) but for now this is >> >> >> >needed >> >> >> >at >> >> >> >least for that use case. >> >> >> > >> >> >> > >> >> >> >As I understand, the conversation here is about trying to take >> >> >> >an >> >> >> >already >> >> >> >populated SD (from another setup I suppose), scanning it and >> >> >> >putting >> >> >> >it >> >> >> >into RHEV? >> >> >> = >> >> >> As I understood it, the purpose of this functionality wasn't to >> >> >> find >> >> >> images which should be removed from storage, but to find images >> >> >> on >> >> >> the >> >> >> domain that oVirt was unaware of and importing them for use >> >> >> (for >> >> >> instance, >> >> >> if a disk was created outside of oVirt on the domain). If one >> >> >> of >> >> >> the >> >> >> use >> >> >> cases for this feature is also the orphaned images mentioned on >> >> >> the >> >> >> feature page, that may expand the functionality into a separate >> >> >> domain >> >> >> scrub and storage import, both of which would call the rescan >> >> >> (meaning the >> >> >> rescan would not actually add to the database, but instead >> >> >> return >> >> >> a >> >> >> list >> >> >> of "orphaned" disk images). >> >> >> = >> >> >> Another solution would be to import all disk images into the >> >> >> database >> >> >> either way, and let the user delete any orphaned images from >> >> >> the >> >> >> GUI. >> >> > >> >> >I think are nice to have, but the problem with the scanning is >> >> >that >> >> >if >> >> >we're not scanning a master domain or an export domain, all we >> >> >will >> >> >see >> >> >is a bunch of images with no context or even hints as to where >> >> >they >> >> >belong. The data that makes it all usable is in the engine >> >> >database >> >> >and >> >> >in the ovf files on the master domain. >> >> > >> >> >This is why I stopped at the orphaned images part of the feature >> >> >- >> >> >because there it's feasible, I would rely on the engine database >> >> >for >> >> >image ID comparisons. >> >> > >> >> >If we present a user with a list of nameless disks, I doubt it >> >> >will >> >> >be of >> >> >any use. >> >> = >> >> The way this would work is by comparing a list of disk images from >> >> vdsm >> >> and from oVirt's database, finding the ones vdsm returns that >> >> oVirt >> >> doesn't have, and then either adding or returning those images. So >> >> oVirt's >> >> db will be used in the comparison. >> > >> >This will work only when scanning storage domains already attached >> >and in >> >use by the current oVirt setup. What I am talking about is what will >> >happen if a LUN that used to be a SD in another oVirt setup is >> >discovered >> >and scanned, with no engine db to compare with. If we don't consider >> >such >> >a use case, life is definitely quite easy, and we're basically >> >within the >> >scope of the orphaned images feature >> = >> This use case should definitely be considered, maybe have a separate >> case >> where the rescan would return all "compatible" disks (i.e. disks that >> aren't just partial snapshots and the like) if the domain has not yet >> been >> mounted. Essentially, it would run the same comparison, but compare >> against an empty list rather than a list of disks. There's no way >> it's as >> simple as that (I'm unsure of the methods oVirt uses to mount a >> domain), >> but it's a good starting point. > >There is no complex method there. For file storage it's just a mount >command, and for block it's LVM (plus iscsi session establishment, if >needed) > >> > >> >> = >> >> As far as presenting the user with nameless disks, that's a point >> >> I >> >> hadn't >> >> considered; we could generate some sort of placeholder metadata >> >> upon >> >> addition to show the user that these are new/orphaned disks that >> >> were >> >> found on the storage domain. Is it safe to assume that the disks >> >> discovered by this feature won't be attached to anything? >> > >> >The oVirt paradigm says "if it isn't in the engine db, it's not >> >ours", so >> >any LV or image we discover that is missing from the DB or the >> >snapshot >> >chain of the image in the DB, is nameless, and orphaned. >> > >> >Such an image on a current SD, belonging to a working oVirt setup is >> >definitely an orphaned image. Attaching these to VMs is usually also >> >useless, because they are more often than not discarded snapshots >> >that >> >didn't get discarded cleanly for some reason. >> > >> > >> >Now, if we want to make this usable, we might want to actually check >> >the >> >qcow2 metadata of the image to see whether it's a mid-chain snapshot >> >(and >> >if so it's probably just a candidate for cleanup), or a standalone >> >qcow2 >> >or raw image, and then we can move on with the virt-* tools, to find >> >out >> >the image size and the filesystems it contains. This will at least >> >provide the user with some usable information about the detected >> >image. >> >If we're talking about scanning an SD that doesn't presently belong >> >to >> >the current oVirt setup, then this is even more relevant, because >> >all of >> >the images will have no VM-related context. >> = >> We're currently working on having disks created outside of the oVirt >> environment, so not all orphaned disks on the existing storage domain >> will >> be artifacts of supposedly-deleted data. > >Do you mean like rhevm-image-upload, or something different? > >> For our use case, disk >> images >> created by us will be able to be imported into oVirt and attached to >> a VM >> created through the engine. Because of this, saying "if it isn't in >> the >> engine db, it's not ours" wouldn't necessarily be true. >> = >> When you talk about checking the metadata, does either oVirt or vdsm >> have >> a simple way to do this? A query of some sort would be ideal for >> this, as >> it could be run for each image as a qualifier for import. > >qemu-img info and libguestfs commands should do. >Besides, our images do come with some metadata (in the LVM tags or a >.meta file) > >> = >> Also, as far as writing the functionality itself, I'm gathering that >> it >> should be structured as a query to return these orphaned images, >> which can >> then be acted upon/added to the database through a separate command >> after >> checking the validity of each image? > >Yes, a simple way to say "import this one to DB, attach to VM X or make >floating", "delete that one", "skip" > > >> > >> >> > = >> >> > >> >> >> > >> >> >> >> = >> >> >> >> > = >> >> >> >> > [1] or a subtab on the storage domain. >> >> >> >> > = >> >> >> >> > _______________________________________________ >> >> >> >> > Engine-devel mailing list >> >> >> >> > Engine-devel(a)ovirt.org >> >> >> >> > http://lists.ovirt.org/mailman/listinfo/engine-devel >> >> >> >> > = >> >> >> >> = >> >> >> > >> >> >> >-- >> >> >> > >> >> >> > >> >> >> > >> >> >> >Regards, >> >> >> > >> >> >> >Dan Yasny >> >> >> >Red Hat Israel >> >> >> >+972 9769 2280 >> >> >> = >> >> >> = >> >> > >> >> >-- >> >> > >> >> > >> >> > >> >> >Regards, >> >> > >> >> >Dan Yasny >> >> >Red Hat Israel >> >> >+972 9769 2280 >> >> = >> >> = >> > >> >-- >> > >> > >> > >> >Regards, >> > >> >Dan Yasny >> >Red Hat Israel >> >+972 9769 2280 >> = >> = > >-- = > > > >Regards, = > >Dan Yasny = >Red Hat Israel = >+972 9769 2280 >_______________________________________________ >Engine-devel mailing list >Engine-devel(a)ovirt.org >http://lists.ovirt.org/mailman/listinfo/engine-devel --===============8565674956477340264==-- From dyasny at redhat.com Thu Aug 2 10:43:55 2012 Content-Type: multipart/mixed; boundary="===============6210416395451061387==" MIME-Version: 1.0 From: Dan Yasny To: devel at ovirt.org Subject: Re: [Engine-devel] Domain rescan action question Date: Thu, 02 Aug 2012 10:43:13 -0400 Message-ID: <1747379863.5617935.1343918593837.JavaMail.root@redhat.com> In-Reply-To: CC4007AD.16A7%rickyh@netapp.com --===============6210416395451061387== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Thanks Ricky, = I've added reviewing this to my todo list Dan ----- Original Message ----- > From: "Ricky Hopper" > To: "Dan Yasny" > Cc: engine-devel(a)ovirt.org > Sent: Thursday, 2 August, 2012 5:37:10 PM > Subject: Re: [Engine-devel] Domain rescan action question > = > In the interest of good discussion, we've put up a feature page for > this > feature (http://wiki.ovirt.org/wiki/Features/Domain_Scan), which > links to > a talk page where modifications can be proposed to how I've laid out > the > feature. So far, it covers how the query works and which commands > will > come about to implement it. I'd appreciate it if anyone concerned > could > check this out and make any changes as they see fit so we can get > going > with the coding. > = > - Ricky > = > On 8/1/12 2:35 PM, "Dan Yasny" wrote: > = > > > > > >----- Original Message ----- > >> From: "Ricky Hopper" > >> To: "Dan Yasny" > >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , > >> "Andrew > >>Cathrow" > >> Sent: Wednesday, 1 August, 2012 8:36:09 PM > >> Subject: Re: [Engine-devel] Domain rescan action question > >> = > >> = > >> = > >> On 8/1/12 10:05 AM, "Dan Yasny" wrote: > >> = > >> > > >> > > >> >----- Original Message ----- > >> >> From: "Ricky Hopper" > >> >> To: "Dan Yasny" > >> >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" , > >> >> "Andrew > >> >>Cathrow" > >> >> Sent: Wednesday, 1 August, 2012 4:56:45 PM > >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> = > >> >> = > >> >> = > >> >> On 8/1/12 9:42 AM, "Dan Yasny" wrote: > >> >> = > >> >> > > >> >> > > >> >> >----- Original Message ----- > >> >> >> From: "Ricky Hopper" > >> >> >> To: "Dan Yasny" , "Andrew Cathrow" > >> >> >> > >> >> >> Cc: engine-devel(a)ovirt.org, "Itamar Heim" > >> >> >> , > >> >> >> "Ricky > >> >> >>Hopper" > >> >> >> Sent: Wednesday, 1 August, 2012 4:34:53 PM > >> >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> >> = > >> >> >> = > >> >> >> = > >> >> >> On 8/1/12 5:59 AM, "Dan Yasny" wrote: > >> >> >> = > >> >> >> > > >> >> >> > > >> >> >> >----- Original Message ----- > >> >> >> >> From: "Andrew Cathrow" > >> >> >> >> To: "Itamar Heim" , "Dan Yasny" > >> >> >> >> , > >> >> >> >>"Ricky Hopper" > >> >> >> >> Cc: engine-devel(a)ovirt.org > >> >> >> >> Sent: Wednesday, 1 August, 2012 12:24:42 AM > >> >> >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> >> >> = > >> >> >> >> = > >> >> >> >> = > >> >> >> >> ----- Original Message ----- > >> >> >> >> > From: "Itamar Heim" > >> >> >> >> > To: "Ricky Hopper" > >> >> >> >> > Cc: engine-devel(a)ovirt.org > >> >> >> >> > Sent: Tuesday, July 31, 2012 4:44:34 PM > >> >> >> >> > Subject: Re: [Engine-devel] Domain rescan action > >> >> >> >> > question > >> >> >> >> > = > >> >> >> >> > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > >> >> >> >> > > Hey all, > >> >> >> >> > > > >> >> >> >> > > As I'm making progress with the domain rescan > >> >> >> >> > > functionality, > >> >> >> >> > > I've > >> >> >> >> > > realized that I'm unsure what to do with any disks > >> >> >> >> > > that > >> >> >> >> > > are > >> >> >> >> > > detected on > >> >> >> >> > > the domain. Should I add them back into the database > >> >> >> >> > > to > >> >> >> >> > > be > >> >> >> >> > > listed > >> >> >> >> > > as > >> >> >> >> > > floating disks, or should I just return a list of > >> >> >> >> > > disk > >> >> >> >> > > images > >> >> >> >> > > to > >> >> >> >> > > be > >> >> >> >> > > attached to whatever the caller of the query needs? > >> >> >> >> > > > >> >> >> >> > > - Ricky > >> >> >> >> > = > >> >> >> >> > i'm not sure they should be added automatically. > >> >> >> >> > I think a dialog[1] showing orphan disks/images on the > >> >> >> >> > storage > >> >> >> >> > domain > >> >> >> >> > for user to choose which to import as 'floating' disks > >> >> >> >> > would > >> >> >> >> > be > >> >> >> >> > better > >> >> >> >> > than auto importing them. > >> >> >> >> > = > >> >> >> >> > there is also the reverse of flagging existing disks as > >> >> >> >> > 'missing' > >> >> >> >> > in > >> >> >> >> > storage? > >> >> >> >> > = > >> >> >> >> = > >> >> >> >> Perhaps we should start a feature page to discuss and > >> >> >> >> better > >> >> >> >> scope > >> >> >> >> it. > >> >> >> >> There is a feature page that we could expand, it doesn't > >> >> >> >> discuss > >> >> >> >> the > >> >> >> >> notion of importing those disks which is certainly > >> >> >> >> something > >> >> >> >> we > >> >> >> >> need > >> >> >> >> to address. > >> >> >> >> = > >> >> >> >> = > >> >> >> >> http://wiki.ovirt.org/wiki/Features/Orphaned_Images > >> >> >> > > >> >> >> >The original idea was to scan the storage domains and > >> >> >> >compare > >> >> >> >the > >> >> >> >images > >> >> >> >lists to the database, thus getting a list of images no > >> >> >> >longer > >> >> >> >relevant > >> >> >> >and scrubbing the storage. This will actually be addressed > >> >> >> >properly > >> >> >> >in > >> >> >> >the future (Ayal can elaborate on that) but for now this is > >> >> >> >needed > >> >> >> >at > >> >> >> >least for that use case. > >> >> >> > > >> >> >> > > >> >> >> >As I understand, the conversation here is about trying to > >> >> >> >take > >> >> >> >an > >> >> >> >already > >> >> >> >populated SD (from another setup I suppose), scanning it > >> >> >> >and > >> >> >> >putting > >> >> >> >it > >> >> >> >into RHEV? > >> >> >> = > >> >> >> As I understood it, the purpose of this functionality wasn't > >> >> >> to > >> >> >> find > >> >> >> images which should be removed from storage, but to find > >> >> >> images > >> >> >> on > >> >> >> the > >> >> >> domain that oVirt was unaware of and importing them for use > >> >> >> (for > >> >> >> instance, > >> >> >> if a disk was created outside of oVirt on the domain). If > >> >> >> one > >> >> >> of > >> >> >> the > >> >> >> use > >> >> >> cases for this feature is also the orphaned images mentioned > >> >> >> on > >> >> >> the > >> >> >> feature page, that may expand the functionality into a > >> >> >> separate > >> >> >> domain > >> >> >> scrub and storage import, both of which would call the > >> >> >> rescan > >> >> >> (meaning the > >> >> >> rescan would not actually add to the database, but instead > >> >> >> return > >> >> >> a > >> >> >> list > >> >> >> of "orphaned" disk images). > >> >> >> = > >> >> >> Another solution would be to import all disk images into the > >> >> >> database > >> >> >> either way, and let the user delete any orphaned images from > >> >> >> the > >> >> >> GUI. > >> >> > > >> >> >I think are nice to have, but the problem with the scanning is > >> >> >that > >> >> >if > >> >> >we're not scanning a master domain or an export domain, all we > >> >> >will > >> >> >see > >> >> >is a bunch of images with no context or even hints as to where > >> >> >they > >> >> >belong. The data that makes it all usable is in the engine > >> >> >database > >> >> >and > >> >> >in the ovf files on the master domain. > >> >> > > >> >> >This is why I stopped at the orphaned images part of the > >> >> >feature > >> >> >- > >> >> >because there it's feasible, I would rely on the engine > >> >> >database > >> >> >for > >> >> >image ID comparisons. > >> >> > > >> >> >If we present a user with a list of nameless disks, I doubt it > >> >> >will > >> >> >be of > >> >> >any use. > >> >> = > >> >> The way this would work is by comparing a list of disk images > >> >> from > >> >> vdsm > >> >> and from oVirt's database, finding the ones vdsm returns that > >> >> oVirt > >> >> doesn't have, and then either adding or returning those images. > >> >> So > >> >> oVirt's > >> >> db will be used in the comparison. > >> > > >> >This will work only when scanning storage domains already > >> >attached > >> >and in > >> >use by the current oVirt setup. What I am talking about is what > >> >will > >> >happen if a LUN that used to be a SD in another oVirt setup is > >> >discovered > >> >and scanned, with no engine db to compare with. If we don't > >> >consider > >> >such > >> >a use case, life is definitely quite easy, and we're basically > >> >within the > >> >scope of the orphaned images feature > >> = > >> This use case should definitely be considered, maybe have a > >> separate > >> case > >> where the rescan would return all "compatible" disks (i.e. disks > >> that > >> aren't just partial snapshots and the like) if the domain has not > >> yet > >> been > >> mounted. Essentially, it would run the same comparison, but > >> compare > >> against an empty list rather than a list of disks. There's no way > >> it's as > >> simple as that (I'm unsure of the methods oVirt uses to mount a > >> domain), > >> but it's a good starting point. > > > >There is no complex method there. For file storage it's just a mount > >command, and for block it's LVM (plus iscsi session establishment, > >if > >needed) > > > >> > > >> >> = > >> >> As far as presenting the user with nameless disks, that's a > >> >> point > >> >> I > >> >> hadn't > >> >> considered; we could generate some sort of placeholder metadata > >> >> upon > >> >> addition to show the user that these are new/orphaned disks > >> >> that > >> >> were > >> >> found on the storage domain. Is it safe to assume that the > >> >> disks > >> >> discovered by this feature won't be attached to anything? > >> > > >> >The oVirt paradigm says "if it isn't in the engine db, it's not > >> >ours", so > >> >any LV or image we discover that is missing from the DB or the > >> >snapshot > >> >chain of the image in the DB, is nameless, and orphaned. > >> > > >> >Such an image on a current SD, belonging to a working oVirt setup > >> >is > >> >definitely an orphaned image. Attaching these to VMs is usually > >> >also > >> >useless, because they are more often than not discarded snapshots > >> >that > >> >didn't get discarded cleanly for some reason. > >> > > >> > > >> >Now, if we want to make this usable, we might want to actually > >> >check > >> >the > >> >qcow2 metadata of the image to see whether it's a mid-chain > >> >snapshot > >> >(and > >> >if so it's probably just a candidate for cleanup), or a > >> >standalone > >> >qcow2 > >> >or raw image, and then we can move on with the virt-* tools, to > >> >find > >> >out > >> >the image size and the filesystems it contains. This will at > >> >least > >> >provide the user with some usable information about the detected > >> >image. > >> >If we're talking about scanning an SD that doesn't presently > >> >belong > >> >to > >> >the current oVirt setup, then this is even more relevant, because > >> >all of > >> >the images will have no VM-related context. > >> = > >> We're currently working on having disks created outside of the > >> oVirt > >> environment, so not all orphaned disks on the existing storage > >> domain > >> will > >> be artifacts of supposedly-deleted data. > > > >Do you mean like rhevm-image-upload, or something different? > > > >> For our use case, disk > >> images > >> created by us will be able to be imported into oVirt and attached > >> to > >> a VM > >> created through the engine. Because of this, saying "if it isn't > >> in > >> the > >> engine db, it's not ours" wouldn't necessarily be true. > >> = > >> When you talk about checking the metadata, does either oVirt or > >> vdsm > >> have > >> a simple way to do this? A query of some sort would be ideal for > >> this, as > >> it could be run for each image as a qualifier for import. > > > >qemu-img info and libguestfs commands should do. > >Besides, our images do come with some metadata (in the LVM tags or a > >.meta file) > > > >> = > >> Also, as far as writing the functionality itself, I'm gathering > >> that > >> it > >> should be structured as a query to return these orphaned images, > >> which can > >> then be acted upon/added to the database through a separate > >> command > >> after > >> checking the validity of each image? > > > >Yes, a simple way to say "import this one to DB, attach to VM X or > >make > >floating", "delete that one", "skip" > > > > > >> > > >> >> > = > >> >> > > >> >> >> > > >> >> >> >> = > >> >> >> >> > = > >> >> >> >> > [1] or a subtab on the storage domain. > >> >> >> >> > = > >> >> >> >> > _______________________________________________ > >> >> >> >> > Engine-devel mailing list > >> >> >> >> > Engine-devel(a)ovirt.org > >> >> >> >> > http://lists.ovirt.org/mailman/listinfo/engine-devel > >> >> >> >> > = > >> >> >> >> = > >> >> >> > > >> >> >> >-- > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> >Regards, > >> >> >> > > >> >> >> >Dan Yasny > >> >> >> >Red Hat Israel > >> >> >> >+972 9769 2280 > >> >> >> = > >> >> >> = > >> >> > > >> >> >-- > >> >> > > >> >> > > >> >> > > >> >> >Regards, > >> >> > > >> >> >Dan Yasny > >> >> >Red Hat Israel > >> >> >+972 9769 2280 > >> >> = > >> >> = > >> > > >> >-- > >> > > >> > > >> > > >> >Regards, > >> > > >> >Dan Yasny > >> >Red Hat Israel > >> >+972 9769 2280 > >> = > >> = > > > >-- > > > > > > > >Regards, > > > >Dan Yasny > >Red Hat Israel > >+972 9769 2280 > >_______________________________________________ > >Engine-devel mailing list > >Engine-devel(a)ovirt.org > >http://lists.ovirt.org/mailman/listinfo/engine-devel > = > = -- = Regards, = Dan Yasny = Red Hat Israel = +972 9769 2280 --===============6210416395451061387==--