[ovirt-users] Re: Backup & Restore

On 20.12.2018 07:41, Yedidyah Bar David wrote: > On Wed, Dec 19, 2018 at 1:35 PM Torsten Stolpmann > <torsten.stolpmann(a)verit.de&gt; wrote: >> >> On 19.12.2018 11:54, Yedidyah Bar David wrote: >>> On Wed, Dec 19, 2018 at 12:34 PM Torsten Stolpmann >>> <torsten.stolpmann(a)verit.de&gt; wrote: >>>> >>>> On 19.12.2018 08:01, Yedidyah Bar David wrote: >>>>> On Tue, Dec 18, 2018 at 5:20 PM Torsten Stolpmann >>>>> <torsten.stolpmann(a)verit.de&gt; wrote: >>>>>> >>>>>> Hi Yedidyah, >>>>>> >>>>>> please find the logs at the following URL: >>>>>> http://www.klaros-testmanagement.com/files/ovirt/ovirt-restore-logs.tar.gz >>>>>> >>>>>> Let me know once you received them safely so I can remove them again. >>>>> >>>>> Done. >>>>> >>>> >>>> Thanks, removed. >>>> >>>>>> >>>>>> I also added the restore.log containing the actual error which occured >>>>>> during the restore. >>>>>> >>>>>> Since this was a clean system I restored on, the setup has been executed >>>>>> after the database restore, so the setup logs probably contain nothing >>>>>> of interest. I added them anyway. >>>>> >>>>> Sorry I wasn't clear enough. I meant the setup logs on the machine used >>>>> to create the backup. So that I can try to see why your backup contained >>>>> this function. Do you still have these by any chance? >>>>> >>>>> Indeed, I can't see anything wrong in the logs above. >>>>> >>>> Sorry for misunderstanding, this totally makes sense. >>>> >>>> I added the setup logs i found at: >>>> http://www.klaros-testmanagement.com/files/ovirt/ovirt-setup-logs.tar.gz >>>> >>>> Again, please let me know once I can remove them. >>> >>> Done. >>> >> Thanks, removed. >> >>>> >>>>>> >>>>>> Please let me know if there is anything I can add to this. >>>>> >>>>> If you do not have setup logs from the original machine, at least try >>>>> to think about its history and tell us notable points - including entire >>>>> version history, setup/upgrade (or similar) problems you had there (and >>>>> perhaps worked around), etc. >>>>> >>>> >>>> We started with one of the first 4.0 releases and updated the system on >>>> a regular basis since then. We almost never skipped a release. >>> >>> The last log there is ovirt-engine-setup-20171231170651-lb3q89.log , >>> meaning it was last upgraded almost a year ago. Were there indeed no >>> more upgrades since? >>> >> >> 20171231170651 is most probably the date when we installed the last >> major release (4.2.0). We did a lot more updates since and I now have a >> suspicion what went wrong here. >> >> We naively assumed that a yum update is sufficient for a minor upgrade >> of the engine installation. > > :-( > >> Rereading the documentation I think we were >> missing explicit engine-setup calls after each minor upgrade. > > Indeed > >> >> It may be the case, that the extra call to engine-setup is required to >> not be part of the yum update. > > engine-setup might need to ask the user questions, so we do not run it > inside yum update. > As long as it clear to users that this is required this is totally ok for me. >> I think in this case it would be a good >> idea to warn users that this step has not yet been taken and the update >> is not completed yet. > > Do you think you would have noticed? It's not that hard to add such a > message during yum update, but not sure it's that helpful. > Yes, I would probably have noticed that. Sooner or later ... :)

> Perhaps we can also make the engine check e.g. if the setup package is > newer than itself, and warn the user in the admin ui. > I think that both is a good idea. Some applications even go the way to lock out users from the GUI until a necessary database migration has been executed by an administrator. Your mileage may vary here but this is the solution I would favor.

>> >> Could this be the cause for the missing logs and behavior discrepancies? >> >> If yes, would another call to engine-setup in the current state fix this >> and allow us to produce correct backups in the future? > > If you still have the source machine, then yes, it should be enough. > Run engine-setup, then engine-backup again to backup, then restore > on the new machine. > Thanks, will do. >> >>> The log indicates the machine was upgraded to ovirt-engine-4.2.0.2 , >>> which didn't remove the uuid functions. The bug I linked at before, >>> 1515635, was fixed in 4.2.1. >>> >>> Based on this, I think the best solution for now would be the patch >>> you suggested. Would you like to open a bug for this and push a patch >>> to gerrit? I can do this as well if you prefer. Bug summary line >>> should probably be something like: >>> >>> engine-setup fails after restoring a backup taken with 4.2.0 >>> >> I currently fear that would only cure the symptom. >> >>> Another solution would be to try using the same engine version during >>> restore (4.2.0.2), and only upgrade later. This is a bit hard, because >>> we do not have separate repos for each version, although they do >>> include all released versions - so you can try e.g.: >>> >>> yum install ovirt-engine-4.2.0.2-1.el7 >>> >>> (meaning, after you remove existing 4.2.7, or you can try yum downgrade). >>> >>> I didn't try this myself, not sure how well it would work. There might >>> be older dependencies to handle, and/or it might break due to too-new >>> stuff. >>> >> I don't think this is necessary. >> >>> Obviously, the best solution is to upgrade more often and backup more >>> often, and have a smaller difference between backup version and restore >>> version, ideally no difference. But I realize this does not always >>> work out for everyone... >>> >> You are right, we did this. >> > > OK. I assume this discussion is theoretical, right? > That you already patched engine-backup manually and restored. Yes it is. Thanks a lot for your patience and support. Torsten > > Best regards, > >> Best regards >> >> Torsten >>>> >>>> There was indeed one glitch during updates which was connected to the >>>> Postgres version in use: >>>> >>>> The initial install was (accidentally) done under Postgres 9.4 which >>>> happened to be present on the machine at that point of time. oVirt 4.0 >>>> was allowing this way back then. I think in 4.1 Postgres 9.2 has been >>>> enforced, so there had been workarounds to allow updates under 9.4. This >>>> got resolved with the migration to 4.2 which brought the Postgres 9.5 >>>> installation (if I recall that correctly). >>>> >>>> Other than that I have no idea which might have introduced this. >>> >>> All of this sounds irrelevant to current problem. Thanks for recalling :-) >>> >>> Best regards, >>> >>>> >>>> If you find something in the logs you need more information on, please >>>> let me know. >>>> >>>> Best regards >>>> >>>> Torsten >>>> >>>> >>>>> If you, or anyone, manages to come up with a flow resulting in a 4.2 >>>>> engine database that contains a function uuid_generate_v1 in the engine >>>>> database schema, I'd definitely want to know about it. >>>>> >>>>> Re-adding others posting in this thread, in case someone has a clue. >>>>> Perhaps one of you has setup logs of the machine on which the backup >>>>> was generated? If so, please share. Thanks. >>>>> >>>>> That said, on a second thought, the implications of this are not that >>>>> significant - mainly somewhat lower performance - so perhaps it's more >>>>> important to fix your bug (by adding a line to IGNORED_ERRORS, as you >>>>> suggested)... but it's still weird. >>>>> >>>>> Thanks and best regards, >>>>> >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Torsten >>>>>> >>>>>> >>>>>> On 18.12.2018 07:54, Yedidyah Bar David wrote: >>>>>>> On Mon, Dec 17, 2018 at 2:26 PM Torsten Stolpmann >>>>>>> <torsten.stolpmann(a)verit.de&gt; wrote: >>>>>>>> >>>>>>>> I experienced the same issue while restoring a full backup (engine & >>>>>>>> dwh) on a clean machine. Both machines are running CentOS 7.6 and >>>>>>>> oVirt 4.2.7. >>>>>>>> >>>>>>>> The issue went away when adding the following line to the IGNORED_ERRORS >>>>>>>> list starting at 1944 in engine-backup: >>>>>>>> >>>>>>>> must be owner of function uuid_generate_v1 >>>>>>>> >>>>>>>> Hope this helps, >>>>>>> >>>>>>> It does, and thanks for the report! >>>>>>> >>>>>>> Can you please share all of your setup logs (/var/log/ovirt-engine/setup/*)? >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> More details: >>>>>>> >>>>>>> This function should not normally be included in a 4.2 backup, and I do >>>>>>> not yet understand why you get this error. >>>>>>> >>>>>>> If it's a clean 4.2 setup, it should never have been defined. >>>>>>> Earlier versions did create it, and the upgrade process should have >>>>>>> dropped it, if it's an upgraded setup. See also: >>>>>>> >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1515635 >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>>> >>>>>>>> Torsten >>>>>>>> >>>>>>>> On 24.05.2018 11:32, emmanuel.thetas(a)obs-nancay.fr wrote: >>>>>>>>> hi, >>>>>>>>> >>>>>>>>> I have the same error when I try to restore on a new hosted VM (ovirt 4.2.3) >>>>>>>>> Have you a solution ? >>>>>>>>> >>>>>>>>> Emmanuel >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YMMFCYUMLCM... >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list -- users(a)ovirt.org >>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FCN5WVAJOGF... >>>>> >>>>> >>>>> >>> >>> >>> > > >