[ovirt-users] I broke my ovirt real good....

[Alexander Wels]

On Friday, April 13, 2018 6:48:31 PM EDT ~Stack~ wrote:
> On 04/13/2018 07:16 AM, Alexander Wels wrote:
> > On Thursday, April 12, 2018 6:26:07 PM EDT ~Stack~ wrote:
> >> Greetings,
> >> 
> >> So I did a over-confident-admin-makes-rookie-mistake. I changed a bunch
> >> of things all back-to-back and thus don't actually know what broke. :-D
> >> 
> >> The only two real "big" changes were:
> >> * Upgrade from 4.2.1 to 4.2.2
> >> * change my ovirtmgmt network
> >> 
> >> The update I followed the upgrade procedures and I thought it all went
> >> pretty well. Because I am moving it from a testing into what I hope will
> >> be a more heavily used environment, I changed my ovirtmgmt network from
> >> 192.168.100.0/24 to 192.168.101.0/24 via the web-gui.
> >> 
> >> That was a touch tricker than just a change as I had to poke the
> >> management engine host to be reachable on both network for a while, then
> >> it just seemed OK.
> >> 
> >> What's happening is:
> >> * I can no longer migrate a vm from one host to the other.
> >> * If I try to do a "reinstall" it dies.
> >> * There is some serious network lag between my hosts on a 10Gb network.
> >> * I've got all kinds of python2.4 failures in my vdsm and mom logs.
> >> 
> >> Those are least the biggies.
> >> 
> >> So while I was planning on moving this to a more active use case, right
> >> now - it is all still my play ground. I would REALLY hate to lose the
> >> VM's but everything else can go and be rebuilt.
> >> 
> >> Given that I've somehow really broke this system pretty good, would it
> >> be more advisable to blow away and rebuild it ALL or can I simply delete
> >> the hypervisor hosts and rebuild them?
> >> 
> >> Thoughts?
> >> 
> >> Thanks!
> >> ~Stack~
> > 
> > As long as you don't destroy the data on your data domain you can rebuild
> > the engine and hosts and then import the existing data domain without too
> > many issues. I have destroyed my engine database many times, and I am
> > still using the same VMs from the same data domain.
> > 
> > Here is what I do when I mess up my database to the point I have to make a
> > new one:
> > 
> > 1. Recreate the engine and database, so that I have basically have an
> > empty
> > engine with no hosts and VMs.
> > 1.1 (Optional) make a new DC that is not default. and add a cluster.
> > 2. Add my hosts (I only have 2 so that is quick and easy).
> > 3. Add a throw away data domain (This is needed to get the DC up so I can
> > import the existing data domain).
> > 4. Import (NOT new, import) the existing data domain.
> > 5. Do to Storage->Storage Domains->VM import and import the VMs I want.
> > 6. Same for templates and disks if needed.
> > 7. After you have imported the VMs/Templates/Disks you can detach and
> > remove the throw away data domain and the one you imported becomes the
> > master domain.
> > 
> > Note if you want to move VMs between your play ground and more serious
> > system you can simply detach your data domain from the play ground, then
> > attach it to the serious engine (so you have 2 engines, one play ground
> > and one serious) and import which VMs you want. That way you won't run
> > into issues with configuring networks and stuff like you experienced.
> 
> Thanks for that help. I did that and everything looks fantastic...except
> I can't migrate VM's. :-/
> 
> It just sits there and in the log files there is the below messages
> repeating. It's like it doesn't care for the fact that this was an
> imported domain or something.
> 
> Thoughts?
> 
> Thanks!
> ~Stack~
> 

Don't know too much about the VDSM side of things. But obviously its looking 
for a storage domain it can't find anymore. You can try restarting VDSM (won't 
affect running VMs) and see if rescans the available storage domains and won't 
try to access it during the migration of the VMs. Other than that I don't 
know.

> 
> 2018-04-13 16:58:59,920-0500 ERROR (monitor/232975a) [storage.Monitor]
> Setting up monitor for 232975ad-1771-4b6b-afda-958f7b745867 failed
> (monitor:329)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
> 326, in _setupLoop
>     self._setupMonitor()
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
> 348, in _setupMonitor
>     self._produceDomain()
>   File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 158, in
> wrapper
>     value = meth(self, *a, **kw)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
> 366, in _produceDomain
>     self.domain = sdCache.produce(self.sdUUID)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110,
> in produce
>     domain.getRealDomain()
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51,
> in getRealDomain
>     return self._cache._realProduce(self._sdUUID)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134,
> in _realProduce
>     domain = self._findDomain(sdUUID)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151,
> in _findDomain
>     return findMethod(sdUUID)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176,
> in _findUnfetchedDomain
>     raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> (u'232975ad-1771-4b6b-afda-958f7b745867',)
> 2018-04-13 16:58:59,923-0500 ERROR (monitor/bc975a4) [storage.Monitor]
> Setting up monitor for bc975a4c-6c38-4248-b3f7-a26945f23693 failed
> (monitor:329)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
> 326, in _setupLoop
>     self._setupMonitor()
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
> 348, in _setupMonitor
>     self._produceDomain()
>   File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 158, in
> wrapper
>     value = meth(self, *a, **kw)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
> 366, in _produceDomain
>     self.domain = sdCache.produce(self.sdUUID)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110,
> in produce
>     domain.getRealDomain()
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51,
> in getRealDomain
>     return self._cache._realProduce(self._sdUUID)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134,
> in _realProduce
>     domain = self._findDomain(sdUUID)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151,
> in _findDomain
>     return findMethod(sdUUID)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176,
> in _findUnfetchedDomain
>     raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> (u'bc975a4c-6c38-4248-b3f7-a26945f23693',)