[ovirt-devel] Re: Testing image transfer and backup with OST environment

On 11/5/20 2:03 PM, Yedidyah Bar David wrote: > On Thu, Nov 5, 2020 at 2:40 PM Marcin Sobczyk <msobczyk(a)redhat.com&gt; wrote: >> >> >> On 11/5/20 1:22 PM, Vojtech Juranek wrote: >>>>> IMO OST should be made easy to interact with from your main development >>>>> machine. >>>> TBH I didn't see much interest in running OST on developers' machines. >>> as it's little bit complex to setup? >> Definitely, that's why the playbook and 'setup_for_ost.sh' was created. >> I hope it will be the long term solution for this problem. >> >>> Making it more easy maybe would increase >>> number of people contributing to OST ... >> But that's a chicken and egg problem - who else is going to contribute >> to OST if not us? >> If we want the setup to be easier, then let's work on it. > I agree, but this requires time. > > Most of the work done on OST so far was for CI. Using it by developers > was a side-interest. > >>>> People are mostly using manual OST runs to verify things and that is what >>>> most of the efforts focus on. It's not that I wouldn't like OST to be more >>>> developer-friendly, I definitely would, but we need more manpower >>>> and interest for that to happen. >>>> >>>>>> I noticed that many of you run OST in a VM ending up with three layers >>>>>> of VMs. >>>>>> I know it works, but I got multiple reports of assertions' timeouts and >>>>>> TBH I just don't >>>>>> see this as a viable solution to work with OST - you need a bare metal >>>>>> for that. >>>>> Why? >>>>> >>>>> After all, we also work on a virtualization product/project. If it's >>>>> not good enough for ourselves, how do we expect others to use it? :-) >>>> I'm really cool with the engine and the hosts being VMs, but deploying >>>> engine and the hosts as VMs nested in other VM is what I think is >>>> unreasonable. >>> I tried this approach in past two days and works fine for me (except the fast >>> it's slow) > (One of the best freudian slips I noticed recently) > >>>> Maybe I'm wrong here, but I don't think our customers run whole oVirt >>>> clusters >>>> inside VMs. There's just too much overhead with all that layers of nesting >>>> and the performance sucks. >>>> >>>>> Also, using bare-metal isn't always that easy/comfortable either, even >>>>> if you have the hardware. >>>> I'm very happy with my servers. What makes working with bm >>>> hard/uncomfortable? >>> IMHO main issue is lack of HW. I really don't want to run it directly on my >>> dev laptop (without running it inside VM). If I ssh/mount FS/tunnel ports to/ >>> from VM or some bare metal server really doesn't matter (assuming reasonable > Lack of HW is definitely an issue, but not the only one. > > When I did use BM for OST at home, my setup was something like this: > > (home router) <-wifi or cable-> (laptop) <-cable-> (switch) <-cable-> BM > > I configured on my laptop PXE for reinstallations. Worked quite well even > if not perfect. > > I worked quite hard on _not_ using my laptop as a router. I ran on it > squid + squid-rpm-cache, and I think I never managed to make OST work > (through a proxy). Eventually I gave up and configured the laptop as a > router, and this worked. > > Before that, I also tried disabling dhcp on the router and running dhcp+dns > PXE on a raspberry pi on the home network, but this was problematic in > other ways (I think the rpi was simply not very stable - had to power- > cycle it every few months). > > I can't use the router as PXE. Hmmm maybe we have a different understanding on "bare metal usage for OST". What I meant is you should use a physical server, install lago et al on it and use it to run OST as usual.

I didn't mean using a separate bare metal for each of the engine and the hosts. That way you don't need to have your own PXE and router - it's taken care of by lago and prebuilt images of VMs.

> >> I agree. I have the privilege of having separate servers to run OST. >> Even though that would work, I can't imagine working with OST >> on a daily basis on my laptop. > I wonder why. Not saying I disagree. OST eats a lot of resources and makes your machine less responsive. Given that, and the lengthy debugging cycles, with a separate server I'm able to work on 2-3 things simultaneously. > > I find the biggest load on my laptop to be from browsers running > web applications. I think this was ok also with my previous laptop, > so assume I can restrict the browsers (with cgroups or whatever) to > only use a slice (and I usually do not really care about their > performance). Just didn't spend the time on trying this yet. > > (Previous laptop was from 2016, 16GB RAM. Current from 2019, 32GB). > >> That also kinda proves my point that people are not being interested >> in running OST on their machines - they don't have machines they could use. >> I see three solutions to this: >> - people start pushing managers to have their own servers > Not very likely. We are supposed to use VMs for that. > >> - we will have a machine-renting solution based on beaker >> (with nice, automatic provisioning for OST etc.), so we can >> work on bare metals > +1 from me. > >> - we focus on the CI and live with the "launch and prey" philosophy :) > If it's fast enough and fully automatic (e.g. using something like Nir's > script, but perhaps with some more features), also ok for me. > > Right now, this is what I use mostly. For things that are much faster > to verify manually, I use manual (non-OST) setup on VMs. > >>> connection speed to bare metal server) >>> >>>> I can think of reprovisioning, but that is not needed for OST usage. >>>> >>>>> CI also uses VMs for this, IIUC. Or did we move there to containers? >>>>> Perhaps we should invest in making this work well inside a container. >>>> CI doesn't use VMs - it uses a mix of containers and bare metals. >>>> The solution for containers can't handle el8 and that's why we're >>>> stuck with running OST on el7 mostly (apart from the aforementioned >>>> bare metals, which use el8). >>>> >>>> There is a 'run-ost-container.sh' script in the project. I think some people >>>> had luck using it, but I never even tried. Again, my personal opinion, as >>>> much as I find containers useful and convenient in different situations, >>>> this is not one of them - you should be using bare metal. >>>> >>>> The "backend for OST" is a subject for a whole, new discussion. >>>> My opinion here is that we should be using oVirt as backend for OST >>>> (as in running oVirt cluster as VMs in oVirt). I'm a big fan of the >>>> dogfooding >>>> concept. > I fully agree. Already talked about this with Sandro some years ago, never > did anything... > >>>> This of course creates a set of new problems like "how can >>>> developers >>>> work with this", "where do we get the hosting oVirt cluster from" etc. >>>> Whooole, new discussion :) > I don't think this is related to whether you use oVirt, or lago/vagrant. > > If every developer has to run an oVirt engine just for running OST inside > it, this indeed has some extra overhead, but if it was the only problem > we'll probably still prefer it compared to re-implementing parts of it > in lago. > > For CI, that's a non-issue, IMO. I'd rather meant having a fixed oVirt installation that's shared by everyone.

In this scenario people would create the OST VMs on that oVirt instance instead of running them locally with lago.

> > Best regards, > >>>> Regards, Marcin >>>> >>>>>> On my bare metal server OST basic run takes 30 mins to complete. This is >>>>>> something one >>>>>> can work with, but we can do even better. >>>>>> >>>>>> Thank you for your input and I hope that we can have more people >>>>>> involved in OST >>>>>> on a regular basis and not once-per-year hackathons. This is a complex >>>>>> project, but it's >>>>>> really useful. >>>>> +1! >>>>> >>>>> Thanks and best regards, >>>>> >>>>>>> Nice. >>>>>>> >>>>>>> Thanks and best regards, >>>>>> [1] >>>>>> https://github.com/lago-project/lago/blob/7bf288ad53da3f1b86c08b3283ee9c5 >>>>>> 118e7605e/lago/providers/libvirt/network.py#L162 [2] >>>>>> https://github.com/oVirt/ovirt-system-tests/blob/6d5c2a0f9fb3c05afc854712 >>>>>> 60065786b5fdc729/ost_utils/ost_utils/pytest/fixtures/engine.py#L105 >