Re: [ovirt-users] Delete Failed to update OVF disks, OVF data isn't updated on those OVF stores (Data Center Default, Storage Domain hostedengine_nfs).

I have added a ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1328718 Looking forward to solve! ( trying to providing as much info as required ). For the short term, wwhat do I need to restore/rollback to get the OVF_STORE back in the Web GUI? is this all db? Op wo 20 apr. 2016 om 09:04 schreef Paul Groeneweg | Pazion <paul(a)pazion.nl > Yes I removed them also from the web interface. > Cen I recreate these or how can I restore? > > Op wo 20 apr. 2016 om 09:01 schreef Roy Golan <rgolan(a)redhat.com> >> On Wed, Apr 20, 2016 at 9:05 AM, Paul Groeneweg | Pazion <paul(a)pazion.nl&gt; >> wrote: >> >>> Hi Roy, >>> >>> What do you mean with a RFE , submit a bug ticket? >>> >>> Yes please. https://*bugzilla*.redhat.com/enter_bug.cgi?product= >> >> *oVirt* >> >> >>> Here is what I did: >>> >>> I removed the OVF disks as explained from the hosted engine/storage. >>> I started another server, tried several things like putting to >>> maintenance and reinstalling, but I keep getting: >>> >>> Apr 20 00:18:00 geisha-3 ovirt-ha-agent: >>> WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable to find >>> OVF_STORE >>> Apr 20 00:18:00 geisha-3 journal: ovirt-ha-agent >>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable >>> to get vm.conf from OVF_STORE, falling back to initial vm.conf >>> Apr 20 00:18:00 geisha-3 ovirt-ha-agent: >>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable >>> to get vm.conf from OVF_STORE, falling back to initial vm.conf >>> Apr 20 00:18:00 geisha-3 journal: ovirt-ha-agent >>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: ''Configuration value >>> not found: file=/var/run/ovirt-hosted-engine-ha/vm.conf, key=memSize'' - >>> trying to restart agent >>> >>> Fact it can't find the OVF store seems logical, but now the >>> /var/run/ovirt-hosted-engine-ha/vm.conf is replace with a file conatining >>> only "None". >>> I tried to set file readonly ( chown root ), but this only threw an >>> error about file not writable, tried different path, but nothing helped. >>> So I am afraid to touch the other running hosts, as same might happen >>> there and I am unable to start hosted engine again. >>> >>> I thought OVF would be created automatically again if it is missing, but >>> it isn't... >>> Can I trigger this OVF, or add it somehow manually? Would deleting the >>> whole hosted_storage trigger an auto import again including OVF? >>> >>> If this provides no solution, I guess, I have to restore the removed OVF >>> store. Would a complete database restore + restoring folder >>> images/<OVF_STORE_ID> be sufficient? >>> Or where is the information about the OVF stores the Web GUI shows >>> stored? >>> >>> >> Did you remove it also from the engine via the webadmin or REST? storage >> tab -> click the hosted_storage domain -> disks subtab -> right click >> remove the failing ovf >> >> >>> Looking forward to resolve this OVF store issue. >>> >>> Thanks in advance!!! >>> >>> >>> >>> Op di 19 apr. 2016 om 10:31 schreef Paul Groeneweg | Pazion < >>> paul(a)pazion.nl>>> >>>> Hi Roy, >>>> >>>> Thanks for this explanation. I will dive into this evening. ( and make >>>> a backup first :-) ) >>>> >>>> Normally the hosted engine only creates 1 ovf disk for the hosted >>>> storage? >>>> >>>> Thanks for the help. >>>> >>>> Op di 19 apr. 2016 om 10:22 schreef Roy Golan <rgolan(a)redhat.com>>>> >>>>> On Mon, Apr 18, 2016 at 10:05 PM, Paul Groeneweg | Pazion < >>>>> paul(a)pazion.nl&gt; wrote: >>>>> >>>>>> I am still wondering about the OVF disk ( and event error ) on my >>>>>> hosted storage domain. >>>>>> >>>>>> My hostedstorage ovf disks ( http://screencast.com/t/AcdqmJWee ) >>>>>> are not being updated ( what I understood is they should be regularly >>>>>> updated ). >>>>>> >>>>>> So I wonder, maybe I can remove these OVF disks and they are >>>>>> recreated automatically? ( Similar when removing the hosted storage domain >>>>>> it was added automatically again ) >>>>>> >>>>>> And for this NFS storage domain, is it normal to have 2 OVF disks? >>>>>> >>>>>> Really looking for a way get these OVF disks right. >>>>>> >>>>>> >>>>>> >>>>> Hi Paul, >>>>> >>>>> What you can do to remove them is to run this sql statement at your >>>>> setup >>>>> >>>>> ```sql >>>>> -- first make sure this is the disk, dates are taken from your >>>>> screenshot >>>>> >>>>> SELECT ovf_disk_id, image_guid, imagestatus, _create_date FROM images, >>>>> storage_domains_ovf_info where ovf_disk_id = images.image_group_id and >>>>> _create_date > '2016-05-01 11:11:29' and _create_date < '2016-05-01 >>>>> 11:11:31'; >>>>> >>>>> -- now delete this disk >>>>> >>>>> DELETE FROM storage_domains_ovf_info where ovf_disk_id = %what was >>>>> found in the last query%' >>>>> ``` >>>>> >>>>> Now you can right-click and remove this disk. >>>>> >>>>> Since the disk of the ovirt-engine resides on the hosted_storage >>>>> domain we can't put this domain into maintenance and fix those kind of >>>>> issues. There for I would like you to kindly open an RFE and mention you're >>>>> scenario so we would supply a way to do this kind of operations in a safe >>>>> way. >>>>> >>>>> Maor thanks for the help and reference. >>>>> >>>>> >>>>>> Op ma 4 apr. 2016 om 09:54 schreef Paul Groeneweg | Pazion < >>>>>> paul(a)pazion.nl>>>>>> >>>>>>> I'd like to add: >>>>>>> >>>>>>> - There are 2 OVF stores in my hosted_storage ( hostedengine_nfs ). >>>>>>> - I checked creation time, they are both created around the same >>>>>>> time http://screencast.com/t/hbXQFlou >>>>>>> >>>>>>> So hopefully there is some way to update hosted storage sp it can be >>>>>>> updated. >>>>>>> >>>>>>> Op do 31 mrt. 2016 om 15:41 schreef Maor Lipchuk < >>>>>>> mlipchuk(a)redhat.com>>>>>>> >>>>>>>> [Adding Roy to the thread] >>>>>>>> Roy, >>>>>>>> >>>>>>>> Can you please share your insight regarding the hosted engine >>>>>>>> behavior. >>>>>>>> It looks that one of the OVF_STORE disks is not valid and I think >>>>>>>> that detach/attach of the storage domain might fix the audit log errors. >>>>>>>> The question is, if it is possible to do so in hosted engine >>>>>>>> environment. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Maor >>>>>>>> >>>>>>>> On Thu, Mar 31, 2016 at 4:14 PM, Paul Groeneweg | Pazion < >>>>>>>> paul(a)pazion.nl&gt; wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> This storage domain is my hosted engine storage domain. So I >>>>>>>>> should put it to maintenance and then detach? >>>>>>>>> http://screencast.com/t/kjgNpI7fQ >>>>>>>>> >>>>>>>>> Am I still able to use the hosed engine ( web interface) when this >>>>>>>>> stoarge domain is in maintenance and detached? >>>>>>>>> >>>>>>>>> As I don't want to risk detaching hosted storage storage domain >>>>>>>>> and as a results breaking my whole setup. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Op do 31 mrt. 2016 om 15:07 schreef Maor Lipchuk < >>>>>>>>> mlipchuk(a)redhat.com>>>>>>>>> >>>>>>>>>> Have you already tried to detach and attach the Storage Domain? >>>>>>>>>> >>>>>>>>>> On Thu, Mar 31, 2016 at 3:11 PM, Paul Groeneweg | Pazion < >>>>>>>>>> paul(a)pazion.nl&gt; wrote: >>>>>>>>>> >>>>>>>>>>> Hi Maor, >>>>>>>>>>> >>>>>>>>>>> I am refering to the eventlog, where these ovf errors appear >>>>>>>>>>> every hour and fill up my eventlog >>>>>>>>>>> >>>>>>>>>>> http://screencast.com/t/S8cfXMsdGM >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Op do 31 mrt. 2016 om 14:07 schreef Maor Lipchuk < >>>>>>>>>>> mlipchuk(a)redhat.com>>>>>>>>>>> >>>>>>>>>>>> Hi Paul, >>>>>>>>>>>> >>>>>>>>>>>> Which problem are you referring, the remove of OVF_STORE disks >>>>>>>>>>>> or the audit log warning? >>>>>>>>>>>> In the screencast I can see that the Storage Domain is active >>>>>>>>>>>> but I didn't notice any audit log errors. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Maor >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 31, 2016 at 2:38 PM, Paul Groeneweg | Pazion < >>>>>>>>>>>> paul(a)pazion.nl&gt; wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Maor, >>>>>>>>>>>>> >>>>>>>>>>>>> The 3.6.4 did not solve the problem. >>>>>>>>>>>>> >>>>>>>>>>>>> Any idea how to fix this issue? >>>>>>>>>>>>> I believe it has something todo with the status of >>>>>>>>>>>>> hosted_storage ( 1st entry ) >>>>>>>>>>>>> http://screencast.com/t/vCx0CQiXm >>>>>>>>>>>>> >>>>>>>>>>>>> Op za 26 mrt. 2016 om 18:08 schreef Maor Lipchuk < >>>>>>>>>>>>> mlipchuk(a)redhat.com>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Paul, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you please update whether the upgrade for 3.6.4 has >>>>>>>>>>>>>> helped. >>>>>>>>>>>>>> Regarding the OVF_STORE disks, those disks should not be >>>>>>>>>>>>>> deleted since deleting them might reflect on the Disaster Recovery scenarios >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Maor >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Mar 24, 2016 at 10:10 PM, Paul Groeneweg | Pazion < >>>>>>>>>>>>>> paul(a)pazion.nl&gt; wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I believe my problem is related to this bug >>>>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1303316 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As you can see in the screenshot the hostedengine storage is >>>>>>>>>>>>>>> unassigned and so both ovf_stores are OK, but not linked and therefore >>>>>>>>>>>>>>> can't be updated?! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So for now I guess I'll wait for update 3.6.4 and cross my >>>>>>>>>>>>>>> fingers and updates solves the event log error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Op do 24 mrt. 2016 om 20:15 schreef Paul Groeneweg | Pazion < >>>>>>>>>>>>>>> paul(a)pazion.nl>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I checked, the OVf, but I can only remove the OVF. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://screencast.com/t/vCx0CQiXm >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What happens when I remove them, is it safe? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I checked agent.log and do not see the errors there.... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>>>> 20:12:28,154::image::116::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) >>>>>>>>>>>>>>>> Preparing images >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>>>> 20:12:28,811::hosted_engine::684::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>>>>>>>>>>>>>>> Reloading vm.conf from the shared storage domain >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>>>> 20:12:28,811::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) >>>>>>>>>>>>>>>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>>>> 20:12:28,936::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>>>>>>>>>>>>>>> Found OVF_STORE: imgUUID:18c50ea6-4654-4525-b241-09e15acf5e99, >>>>>>>>>>>>>>>> volUUID:2f2ccb59-a3f3-43bf-87eb-53595af01cf5 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>>>> 20:12:29,147::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>>>>>>>>>>>>>>> Found OVF_STORE: imgUUID:6e14348b-af7a-49bc-9af2-8b703c17a53d, >>>>>>>>>>>>>>>> volUUID:fabdd6f4-b8d6-4ffe-889c-df86b34619ca >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>>>> 20:12:29,420::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>>>>>>>>>>>>>>> Extracting Engine VM OVF from the OVF_STORE >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>>>> 20:12:29,580::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>>>>>>>>>>>>>>> OVF_STORE volume path: /rhev/data-center/mnt/hostedstorage.pazion.nl: >>>>>>>>>>>>>>>> _opt_hosted-engine/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/images/6e14348b-af7a-49bc-9af2-8b703c17a53d/fabdd6f4-b8d6-4ffe-889c-df86b34619ca >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>>>> 20:12:29,861::config::225::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) >>>>>>>>>>>>>>>> Found an OVF for HE VM, trying to convert >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>>>> 20:12:29,865::config::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) >>>>>>>>>>>>>>>> Got vm.conf from OVF_STORE >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>>>> 20:12:29,997::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>>>>>>>>>>>>>>> Current state EngineUp (score: 3400) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So leaves me wondering if I should worry about the errors >>>>>>>>>>>>>>>> in the event log. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Op do 24 mrt. 2016 om 16:18 schreef Paul Groeneweg | Pazion >>>>>>>>>>>>>>>> <paul(a)pazion.nl>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> These OVF stores are created on my hosted-engine storage >>>>>>>>>>>>>>>>> instance. I did not found any reference in the hosted-engine.conf, so you >>>>>>>>>>>>>>>>> are sure they can't be deleted? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So it holds only info about the hosted-engine disk? So >>>>>>>>>>>>>>>>> when detaching, do I have any risk destroying my hosted-engine? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I can just detach them in this screen: >>>>>>>>>>>>>>>>> http://screencast.com/t/ymnzsNHj7e and then re-attach? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I check file permissions, but this looked good compared to >>>>>>>>>>>>>>>>> the other images. So really strange this eventlog. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> Paul >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Op do 24 mrt. 2016 om 10:01 schreef Maor Lipchuk < >>>>>>>>>>>>>>>>> mlipchuk(a)redhat.com>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Met vriendelijke groeten, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Paul Groeneweg >>>>>>>>>>>>>>>>>> Pazion >>>>>>>>>>>>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> T +31 26 3020038 >>>>>>>>>>>>>>>>>> M +31 614 277 577 >>>>>>>>>>>>>>>>>> E paul(a)pazion.nl >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ***disclaimer*** >>>>>>>>>>>>>>>>>> "This e-mail and any attachments thereto may contain >>>>>>>>>>>>>>>>>> information which is confidential and/or protected by intellectual property >>>>>>>>>>>>>>>>>> rights and are intended for the sole use of the recipient(s) named above. >>>>>>>>>>>>>>>>>> Any use of the information contained herein (including, but not limited to, >>>>>>>>>>>>>>>>>> total or partial reproduction, communication or distribution in any form) >>>>>>>>>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. If you >>>>>>>>>>>>>>>>>> have received this e-mail in error, please notify the sender either by >>>>>>>>>>>>>>>>>> telephone or by e-mail and delete the material from any computer. Thank you >>>>>>>>>>>>>>>>>> for your cooperation." >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, Mar 24, 2016 at 12:12 AM, Paul Groeneweg | Pazion >>>>>>>>>>>>>>>>>> <paul(a)pazion.nl&gt; wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> After the 3.6 updates ( which didn't went without a >>>>>>>>>>>>>>>>>>> hitch ) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I get the following errors in my event log: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Failed to update OVF disks >>>>>>>>>>>>>>>>>>> 18c50ea6-4654-4525-b241-09e15acf5e99, OVF data isn't updated on those OVF >>>>>>>>>>>>>>>>>>> stores (Data Center Default, Storage Domain hostedengine_nfs). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> VDSM command failed: Could not acquire resource. >>>>>>>>>>>>>>>>>>> Probably resource factory threw an exception.: () >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> http://screencast.com/t/S8cfXMsdGM >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> When I check on file there is some data, but not >>>>>>>>>>>>>>>>>>> updated: http://screencast.com/t/hbXQFlou >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> When I check in the web interface I see 2 OVF files >>>>>>>>>>>>>>>>>>> listed. What are these for, can I delete them? >>>>>>>>>>>>>>>>>>> http://screencast.com/t/ymnzsNHj7e >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hopefully someone knows what to do about these >>>>>>>>>>>>>>>>>>> warnings/erros and whether I can delete the OVF files. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>> Paul Groeneweg >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Paul, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The OVF_STORE disks are disks which preserve all the VMs >>>>>>>>>>>>>>>>>> and Templates OVF data and are mostly use for disaster recovery scenarios. >>>>>>>>>>>>>>>>>> Those disks can not be deleted. >>>>>>>>>>>>>>>>>> Regarding the audit log which you got, can you try to >>>>>>>>>>>>>>>>>> detach and attach the Storage once again and let me know if you still get >>>>>>>>>>>>>>>>>> this even log. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>> Maor >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>