The logs are not from the machine where the hosted engine is running on, but from the SPM. Op wo 20 apr. 2016 om 17:19 schreef Paul Groeneweg | Pazion <paul@pazion.nl
:
Hereby the logs.
Op wo 20 apr. 2016 om 17:11 schreef Maor Lipchuk <mlipchuk@redhat.com>:
Hi Paul,
Can u please attach the engine and VDSM logs with those failures to check the origin of those failures
Thanks, Maor
On Wed, Apr 20, 2016 at 6:06 PM, Paul Groeneweg | Pazion <paul@pazion.nl> wrote:
Looks like the system does try recreate the OVF :-) Too bad this failed again...
http://screencast.com/t/RlYCR1rk8T http://screencast.com/t/CpcQuoKg
Failed to create OVF store disk for Storage Domain hostedengine_nfs. The Disk with the id b6f34661-8701-4f82-a07c-ed7faab4a1b8 might be removed manually for automatic attempt to create new one. OVF updates won't be attempted on the created disk.
And on the hosted storage disk tab : http://screencast.com/t/ZmwjsGoQ1Xbp
Op wo 20 apr. 2016 om 09:17 schreef Paul Groeneweg | Pazion < paul@pazion.nl>:
I have added a ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1328718
Looking forward to solve! ( trying to providing as much info as required ).
For the short term, wwhat do I need to restore/rollback to get the OVF_STORE back in the Web GUI? is this all db?
Op wo 20 apr. 2016 om 09:04 schreef Paul Groeneweg | Pazion < paul@pazion.nl>:
Yes I removed them also from the web interface. Cen I recreate these or how can I restore?
Op wo 20 apr. 2016 om 09:01 schreef Roy Golan <rgolan@redhat.com>:
On Wed, Apr 20, 2016 at 9:05 AM, Paul Groeneweg | Pazion < paul@pazion.nl> wrote:
> Hi Roy, > > What do you mean with a RFE , submit a bug ticket? > > Yes please. https://*bugzilla*.redhat.com/enter_bug.cgi?product=
*oVirt*
> Here is what I did: > > I removed the OVF disks as explained from the hosted engine/storage. > I started another server, tried several things like putting to > maintenance and reinstalling, but I keep getting: > > Apr 20 00:18:00 geisha-3 ovirt-ha-agent: > WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable to find > OVF_STORE > Apr 20 00:18:00 geisha-3 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable > to get vm.conf from OVF_STORE, falling back to initial vm.conf > Apr 20 00:18:00 geisha-3 ovirt-ha-agent: > ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable > to get vm.conf from OVF_STORE, falling back to initial vm.conf > Apr 20 00:18:00 geisha-3 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: ''Configuration value > not found: file=/var/run/ovirt-hosted-engine-ha/vm.conf, key=memSize'' - > trying to restart agent > > Fact it can't find the OVF store seems logical, but now the > /var/run/ovirt-hosted-engine-ha/vm.conf is replace with a file conatining > only "None". > I tried to set file readonly ( chown root ), but this only threw an > error about file not writable, tried different path, but nothing helped. > So I am afraid to touch the other running hosts, as same might > happen there and I am unable to start hosted engine again. > > I thought OVF would be created automatically again if it is missing, > but it isn't... > Can I trigger this OVF, or add it somehow manually? Would deleting > the whole hosted_storage trigger an auto import again including OVF? > > If this provides no solution, I guess, I have to restore the removed > OVF store. Would a complete database restore + restoring folder > images/<OVF_STORE_ID> be sufficient? > Or where is the information about the OVF stores the Web GUI shows > stored? > > Did you remove it also from the engine via the webadmin or REST? storage tab -> click the hosted_storage domain -> disks subtab -> right click remove the failing ovf
> Looking forward to resolve this OVF store issue. > > Thanks in advance!!! > > > > Op di 19 apr. 2016 om 10:31 schreef Paul Groeneweg | Pazion < > paul@pazion.nl>: > >> Hi Roy, >> >> Thanks for this explanation. I will dive into this evening. ( and >> make a backup first :-) ) >> >> Normally the hosted engine only creates 1 ovf disk for the hosted >> storage? >> >> Thanks for the help. >> >> Op di 19 apr. 2016 om 10:22 schreef Roy Golan <rgolan@redhat.com>: >> >>> On Mon, Apr 18, 2016 at 10:05 PM, Paul Groeneweg | Pazion < >>> paul@pazion.nl> wrote: >>> >>>> I am still wondering about the OVF disk ( and event error ) on my >>>> hosted storage domain. >>>> >>>> My hostedstorage ovf disks ( http://screencast.com/t/AcdqmJWee ) >>>> are not being updated ( what I understood is they should be regularly >>>> updated ). >>>> >>>> So I wonder, maybe I can remove these OVF disks and they are >>>> recreated automatically? ( Similar when removing the hosted storage domain >>>> it was added automatically again ) >>>> >>>> And for this NFS storage domain, is it normal to have 2 OVF disks? >>>> >>>> Really looking for a way get these OVF disks right. >>>> >>>> >>>> >>> Hi Paul, >>> >>> What you can do to remove them is to run this sql statement at >>> your setup >>> >>> ```sql >>> -- first make sure this is the disk, dates are taken from your >>> screenshot >>> >>> SELECT ovf_disk_id, image_guid, imagestatus, _create_date FROM >>> images, storage_domains_ovf_info where ovf_disk_id = images.image_group_id >>> and _create_date > '2016-05-01 11:11:29' and _create_date < '2016-05-01 >>> 11:11:31'; >>> >>> -- now delete this disk >>> >>> DELETE FROM storage_domains_ovf_info where ovf_disk_id = %what >>> was found in the last query%' >>> ``` >>> >>> Now you can right-click and remove this disk. >>> >>> Since the disk of the ovirt-engine resides on the hosted_storage >>> domain we can't put this domain into maintenance and fix those kind of >>> issues. There for I would like you to kindly open an RFE and mention you're >>> scenario so we would supply a way to do this kind of operations in a safe >>> way. >>> >>> Maor thanks for the help and reference. >>> >>> >>>> Op ma 4 apr. 2016 om 09:54 schreef Paul Groeneweg | Pazion < >>>> paul@pazion.nl>: >>>> >>>>> I'd like to add: >>>>> >>>>> - There are 2 OVF stores in my hosted_storage ( hostedengine_nfs >>>>> ). >>>>> - I checked creation time, they are both created around the same >>>>> time http://screencast.com/t/hbXQFlou >>>>> >>>>> So hopefully there is some way to update hosted storage sp it >>>>> can be updated. >>>>> >>>>> Op do 31 mrt. 2016 om 15:41 schreef Maor Lipchuk < >>>>> mlipchuk@redhat.com>: >>>>> >>>>>> [Adding Roy to the thread] >>>>>> Roy, >>>>>> >>>>>> Can you please share your insight regarding the hosted engine >>>>>> behavior. >>>>>> It looks that one of the OVF_STORE disks is not valid and I >>>>>> think that detach/attach of the storage domain might fix the audit log >>>>>> errors. >>>>>> The question is, if it is possible to do so in hosted engine >>>>>> environment. >>>>>> >>>>>> Regards, >>>>>> Maor >>>>>> >>>>>> On Thu, Mar 31, 2016 at 4:14 PM, Paul Groeneweg | Pazion < >>>>>> paul@pazion.nl> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> This storage domain is my hosted engine storage domain. So I >>>>>>> should put it to maintenance and then detach? >>>>>>> http://screencast.com/t/kjgNpI7fQ >>>>>>> >>>>>>> Am I still able to use the hosed engine ( web interface) when >>>>>>> this stoarge domain is in maintenance and detached? >>>>>>> >>>>>>> As I don't want to risk detaching hosted storage storage >>>>>>> domain and as a results breaking my whole setup. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Op do 31 mrt. 2016 om 15:07 schreef Maor Lipchuk < >>>>>>> mlipchuk@redhat.com>: >>>>>>> >>>>>>>> Have you already tried to detach and attach the Storage >>>>>>>> Domain? >>>>>>>> >>>>>>>> On Thu, Mar 31, 2016 at 3:11 PM, Paul Groeneweg | Pazion < >>>>>>>> paul@pazion.nl> wrote: >>>>>>>> >>>>>>>>> Hi Maor, >>>>>>>>> >>>>>>>>> I am refering to the eventlog, where these ovf errors appear >>>>>>>>> every hour and fill up my eventlog >>>>>>>>> >>>>>>>>> http://screencast.com/t/S8cfXMsdGM >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Op do 31 mrt. 2016 om 14:07 schreef Maor Lipchuk < >>>>>>>>> mlipchuk@redhat.com>: >>>>>>>>> >>>>>>>>>> Hi Paul, >>>>>>>>>> >>>>>>>>>> Which problem are you referring, the remove of OVF_STORE >>>>>>>>>> disks or the audit log warning? >>>>>>>>>> In the screencast I can see that the Storage Domain is >>>>>>>>>> active but I didn't notice any audit log errors. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Maor >>>>>>>>>> >>>>>>>>>> On Thu, Mar 31, 2016 at 2:38 PM, Paul Groeneweg | Pazion < >>>>>>>>>> paul@pazion.nl> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Maor, >>>>>>>>>>> >>>>>>>>>>> The 3.6.4 did not solve the problem. >>>>>>>>>>> >>>>>>>>>>> Any idea how to fix this issue? >>>>>>>>>>> I believe it has something todo with the status of >>>>>>>>>>> hosted_storage ( 1st entry ) >>>>>>>>>>> http://screencast.com/t/vCx0CQiXm >>>>>>>>>>> >>>>>>>>>>> Op za 26 mrt. 2016 om 18:08 schreef Maor Lipchuk < >>>>>>>>>>> mlipchuk@redhat.com>: >>>>>>>>>>> >>>>>>>>>>>> Hi Paul, >>>>>>>>>>>> >>>>>>>>>>>> Can you please update whether the upgrade for 3.6.4 has >>>>>>>>>>>> helped. >>>>>>>>>>>> Regarding the OVF_STORE disks, those disks should not be >>>>>>>>>>>> deleted since deleting them might reflect on the Disaster Recovery scenarios >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Maor >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 24, 2016 at 10:10 PM, Paul Groeneweg | Pazion >>>>>>>>>>>> <paul@pazion.nl> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I believe my problem is related to this bug >>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1303316 >>>>>>>>>>>>> >>>>>>>>>>>>> As you can see in the screenshot the hostedengine >>>>>>>>>>>>> storage is unassigned and so both ovf_stores are OK, but not linked and >>>>>>>>>>>>> therefore can't be updated?! >>>>>>>>>>>>> >>>>>>>>>>>>> So for now I guess I'll wait for update 3.6.4 and cross >>>>>>>>>>>>> my fingers and updates solves the event log error. >>>>>>>>>>>>> >>>>>>>>>>>>> Op do 24 mrt. 2016 om 20:15 schreef Paul Groeneweg | >>>>>>>>>>>>> Pazion <paul@pazion.nl>: >>>>>>>>>>>>> >>>>>>>>>>>>>> I checked, the OVf, but I can only remove the OVF. >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://screencast.com/t/vCx0CQiXm >>>>>>>>>>>>>> >>>>>>>>>>>>>> What happens when I remove them, is it safe? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I checked agent.log and do not see the errors there.... >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>> 20:12:28,154::image::116::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) >>>>>>>>>>>>>> Preparing images >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>> 20:12:28,811::hosted_engine::684::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>>>>>>>>>>>>> Reloading vm.conf from the shared storage domain >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>> 20:12:28,811::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) >>>>>>>>>>>>>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>> 20:12:28,936::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>>>>>>>>>>>>> Found OVF_STORE: imgUUID:18c50ea6-4654-4525-b241-09e15acf5e99, >>>>>>>>>>>>>> volUUID:2f2ccb59-a3f3-43bf-87eb-53595af01cf5 >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>> 20:12:29,147::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>>>>>>>>>>>>> Found OVF_STORE: imgUUID:6e14348b-af7a-49bc-9af2-8b703c17a53d, >>>>>>>>>>>>>> volUUID:fabdd6f4-b8d6-4ffe-889c-df86b34619ca >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>> 20:12:29,420::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>>>>>>>>>>>>> Extracting Engine VM OVF from the OVF_STORE >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>> 20:12:29,580::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>>>>>>>>>>>>> OVF_STORE volume path: /rhev/data-center/mnt/hostedstorage.pazion.nl: >>>>>>>>>>>>>> _opt_hosted-engine/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/images/6e14348b-af7a-49bc-9af2-8b703c17a53d/fabdd6f4-b8d6-4ffe-889c-df86b34619ca >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>> 20:12:29,861::config::225::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) >>>>>>>>>>>>>> Found an OVF for HE VM, trying to convert >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>> 20:12:29,865::config::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) >>>>>>>>>>>>>> Got vm.conf from OVF_STORE >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2016-03-24 >>>>>>>>>>>>>> 20:12:29,997::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>>>>>>>>>>>>> Current state EngineUp (score: 3400) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> So leaves me wondering if I should worry about the >>>>>>>>>>>>>> errors in the event log. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Op do 24 mrt. 2016 om 16:18 schreef Paul Groeneweg | >>>>>>>>>>>>>> Pazion <paul@pazion.nl>: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These OVF stores are created on my hosted-engine >>>>>>>>>>>>>>> storage instance. I did not found any reference in the hosted-engine.conf, >>>>>>>>>>>>>>> so you are sure they can't be deleted? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So it holds only info about the hosted-engine disk? So >>>>>>>>>>>>>>> when detaching, do I have any risk destroying my hosted-engine? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I can just detach them in this screen: >>>>>>>>>>>>>>> http://screencast.com/t/ymnzsNHj7e and then re-attach? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I check file permissions, but this looked good >>>>>>>>>>>>>>> compared to the other images. So really strange this eventlog. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Paul >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Op do 24 mrt. 2016 om 10:01 schreef Maor Lipchuk < >>>>>>>>>>>>>>> mlipchuk@redhat.com>: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Met vriendelijke groeten, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Paul Groeneweg >>>>>>>>>>>>>>>> Pazion >>>>>>>>>>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> T +31 26 3020038 >>>>>>>>>>>>>>>> M +31 614 277 577 >>>>>>>>>>>>>>>> E paul@pazion.nl >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ***disclaimer*** >>>>>>>>>>>>>>>> "This e-mail and any attachments thereto may contain >>>>>>>>>>>>>>>> information which is confidential and/or protected by intellectual property >>>>>>>>>>>>>>>> rights and are intended for the sole use of the recipient(s) named above. >>>>>>>>>>>>>>>> Any use of the information contained herein (including, but not limited to, >>>>>>>>>>>>>>>> total or partial reproduction, communication or distribution in any form) >>>>>>>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. If you >>>>>>>>>>>>>>>> have received this e-mail in error, please notify the sender either by >>>>>>>>>>>>>>>> telephone or by e-mail and delete the material from any computer. Thank you >>>>>>>>>>>>>>>> for your cooperation." >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, Mar 24, 2016 at 12:12 AM, Paul Groeneweg | >>>>>>>>>>>>>>>> Pazion <paul@pazion.nl> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> After the 3.6 updates ( which didn't went without a >>>>>>>>>>>>>>>>> hitch ) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I get the following errors in my event log: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Failed to update OVF disks >>>>>>>>>>>>>>>>> 18c50ea6-4654-4525-b241-09e15acf5e99, OVF data isn't updated on those OVF >>>>>>>>>>>>>>>>> stores (Data Center Default, Storage Domain hostedengine_nfs). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> VDSM command failed: Could not acquire resource. >>>>>>>>>>>>>>>>> Probably resource factory threw an exception.: () >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://screencast.com/t/S8cfXMsdGM >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> When I check on file there is some data, but not >>>>>>>>>>>>>>>>> updated: http://screencast.com/t/hbXQFlou >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> When I check in the web interface I see 2 OVF files >>>>>>>>>>>>>>>>> listed. What are these for, can I delete them? >>>>>>>>>>>>>>>>> http://screencast.com/t/ymnzsNHj7e >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hopefully someone knows what to do about these >>>>>>>>>>>>>>>>> warnings/erros and whether I can delete the OVF files. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>> Paul Groeneweg >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Paul, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The OVF_STORE disks are disks which preserve all the >>>>>>>>>>>>>>>> VMs and Templates OVF data and are mostly use for disaster recovery >>>>>>>>>>>>>>>> scenarios. >>>>>>>>>>>>>>>> Those disks can not be deleted. >>>>>>>>>>>>>>>> Regarding the audit log which you got, can you try to >>>>>>>>>>>>>>>> detach and attach the Storage once again and let me know if you still get >>>>>>>>>>>>>>>> this even log. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> Maor >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>