I broke my ovirt real good....
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --NWB25NR97RjNKvu7NPH6FwJ2Z6BILVPLZ Content-Type: multipart/mixed; boundary="xECPZYKnQRjXfho4Fx0Q1QroKLPENPCQr"; protected-headers="v1" From: ~Stack~ <i.am.stack@gmail.com> To: users <users@ovirt.org> Message-ID: <816729cd-1ec6-f2e6-6986-2a18e83ab650@gmail.com> Subject: I broke my ovirt real good.... --xECPZYKnQRjXfho4Fx0Q1QroKLPENPCQr Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Greetings, So I did a over-confident-admin-makes-rookie-mistake. I changed a bunch of things all back-to-back and thus don't actually know what broke. :-D The only two real "big" changes were: * Upgrade from 4.2.1 to 4.2.2 * change my ovirtmgmt network The update I followed the upgrade procedures and I thought it all went pretty well. Because I am moving it from a testing into what I hope will be a more heavily used environment, I changed my ovirtmgmt network from 192.168.100.0/24 to 192.168.101.0/24 via the web-gui. That was a touch tricker than just a change as I had to poke the management engine host to be reachable on both network for a while, then it just seemed OK. What's happening is: * I can no longer migrate a vm from one host to the other. * If I try to do a "reinstall" it dies. * There is some serious network lag between my hosts on a 10Gb network. * I've got all kinds of python2.4 failures in my vdsm and mom logs. Those are least the biggies. So while I was planning on moving this to a more active use case, right now - it is all still my play ground. I would REALLY hate to lose the VM's but everything else can go and be rebuilt. Given that I've somehow really broke this system pretty good, would it be more advisable to blow away and rebuild it ALL or can I simply delete the hypervisor hosts and rebuild them? Thoughts? Thanks! ~Stack~ --xECPZYKnQRjXfho4Fx0Q1QroKLPENPCQr-- --NWB25NR97RjNKvu7NPH6FwJ2Z6BILVPLZ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJaz9z/AAoJELkej+ysXJPmGWoP/1UVu8ui41H4mSs0SKBRu+Dp 3LwJxzweGlp4WnJ96aLeC/Wy+rLM35O3V/KmtetLIjqJksInlXn5ohTQKDFSLJqI y0Q15N+QNxixPadWeTT5qwBzjhjgySSbMyYdhdh6Wzr4wdVhSURNMgD8B73uaeAC K+yqTAckveBx6OcT0FRcYR7cCRYhrBdERHUHysA1RX0wnkxzxYnEvtNq9/cS9+tr lORnVWsnfm/d4xQgwztXRBwBKm0KOJ48J0gyG+dZv9dUTCGAdP2jz30ie+2JrQu6 mYy0nFKA9HaFY7+5p/A3J0KMjCkMAKmWIYJLo4zQSn9x289omNVnIZ+Ow1GgHtO8 uddRN/zgDj8gaWDH9WfirzHQ+eiI4hu8JWt3wBGdwR0Zv/wVInyS/Xi2TvKyJ1Hq /rX0Xa13KqtfIVKj73v17iyA0r/BalBrldgSHIA+umCM3hdAgeXCTTDYRgpLFNOV Ci7WkvYXOhTFWFPpmHCU9Lv35v0HhcdDILgPTkb/1gLIkCbh6dWuViL8N9Ok3jxM E3y9aLLKefa6C7mL6Mc862sOenZ7AAFbNQl85NNwhW5IDT7jHn6xKK8O+ARvQ7fC UqUkQgfrpi8OAVJHoD+lmJV+NgZT3sTORxZROtJrak3AsgKyITHAL/mtqOjZxcNT 1qpLw6Hf/yddk5Ler8Pf =y3f/ -----END PGP SIGNATURE----- --NWB25NR97RjNKvu7NPH6FwJ2Z6BILVPLZ--
Hi Stack, Do you use FQDN? Did you perhaps hit this one https://www.ovirt.org/blog/2016/05/modify-ifcfg-files/ ? The discussion in this bug report may be of assistance in that case: https://bugzilla.redhat.com/show_bug.cgi?id=1252534 If you've stored the VM disks and templates and whatnot on a network share like NFS, you should be able to start all over and import your old (current) storage domains and start using your templates etc. Best wishes // Mike -----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of ~Stack~ Sent: 13. april 2018 00:26 To: users <users@ovirt.org> Subject: [ovirt-users] I broke my ovirt real good.... Greetings, So I did a over-confident-admin-makes-rookie-mistake. I changed a bunch of things all back-to-back and thus don't actually know what broke. :-D The only two real "big" changes were: * Upgrade from 4.2.1 to 4.2.2 * change my ovirtmgmt network The update I followed the upgrade procedures and I thought it all went pretty well. Because I am moving it from a testing into what I hope will be a more heavily used environment, I changed my ovirtmgmt network from 192.168.100.0/24 to 192.168.101.0/24 via the web-gui. That was a touch tricker than just a change as I had to poke the management engine host to be reachable on both network for a while, then it just seemed OK. What's happening is: * I can no longer migrate a vm from one host to the other. * If I try to do a "reinstall" it dies. * There is some serious network lag between my hosts on a 10Gb network. * I've got all kinds of python2.4 failures in my vdsm and mom logs. Those are least the biggies. So while I was planning on moving this to a more active use case, right now - it is all still my play ground. I would REALLY hate to lose the VM's but everything else can go and be rebuilt. Given that I've somehow really broke this system pretty good, would it be more advisable to blow away and rebuild it ALL or can I simply delete the hypervisor hosts and rebuild them? Thoughts? Thanks! ~Stack~
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --BVmfCuhTspWg090zQKOmgokeery8TiAuX Content-Type: multipart/mixed; boundary="V8AnLSxbk5dQoPXpEqjsmJ3HofOKOTeEA"; protected-headers="v1" From: ~Stack~ <i.am.stack@gmail.com> To: users <users@ovirt.org> Message-ID: <dada8a96-a7b7-af2d-1e31-86c88baeb76d@gmail.com> Subject: Re: [ovirt-users] I broke my ovirt real good.... References: <816729cd-1ec6-f2e6-6986-2a18e83ab650@gmail.com> <48ea2d3977b34cbf92602811be1f01cc@KBNMXEXC01.Demant.com> In-Reply-To: <48ea2d3977b34cbf92602811be1f01cc@KBNMXEXC01.Demant.com> --V8AnLSxbk5dQoPXpEqjsmJ3HofOKOTeEA Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 04/13/2018 03:02 AM, Michael Mortensen (MCMR) wrote:
That looks very interesting! I will investigate that.
I am currently using NFS. I will see how this networking issue you pointed me to works out, then maybe rebuild. Thank you for the assistance! ~Stack~ --V8AnLSxbk5dQoPXpEqjsmJ3HofOKOTeEA-- --BVmfCuhTspWg090zQKOmgokeery8TiAuX Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJa0I1VAAoJELkej+ysXJPmjMMP/3H/+FGTDNCjbgZuW8jWTmWl u1JqnuqsJ6CgQTlWemLZh3yN8z2Xf30mZVSra3oUaYDNF4vruGzQjdJyPf57/Bny ZyPSKVSwufCnVlF3wSDF9xBfRkuAZR1jfilXa5ydzIM6DGIN7YB4PFumAFNby/aL yaV0Dr6vQ4sE4TMkpGKpfVgvbR4UbkkWAoPgJQZKPSEwtXmADhpdlxEW6kD9DUPY TIsHBs/Bd1+55CFTt3Pl51sXdRUdqsQWl/LLRL/mX6idfB6q4BFbg2V2X5d7DmU2 udTQvTDdj5fj5NEKwkfwwJx0Ia4Mxpf3V6BvcszgAmUvje0/ENM/vR9IvpPKpbHk Wn+qV/XN/nM2Z0GT4wldIzDurqwtsXyoIncqUasq7KIhy4ujHvHD/65Vr0Kx6XCJ o+PKpbwx9hVg3fZ6tE7tOqO0jWrRMZyCc26XwN2W7s/bwEGikvzFGq/4xpD7LcrE LCMKd1/lw3m2EAJynaIn2SmEqNneCYtzgRhLg5kKPtiMUb3v0nZYqr43ZLySlBxv xsDhYUl01o3FaJlG9z+jneK4YavxDGjx93rrQfhSBx/eOgXIT9tKrDPKIxOy4aay rn0TcVWwTZnr+wOc9u23OYOo0yJF9UxGlRMjz8lbXXUEOOWhm5DuKpKjiNzKV0Vq TTg71pF5gPjj38xEP4HR =8BVe -----END PGP SIGNATURE----- --BVmfCuhTspWg090zQKOmgokeery8TiAuX--
On Thursday, April 12, 2018 6:26:07 PM EDT ~Stack~ wrote:
As long as you don't destroy the data on your data domain you can rebuild the engine and hosts and then import the existing data domain without too many issues. I have destroyed my engine database many times, and I am still using the same VMs from the same data domain. Here is what I do when I mess up my database to the point I have to make a new one: 1. Recreate the engine and database, so that I have basically have an empty engine with no hosts and VMs. 1.1 (Optional) make a new DC that is not default. and add a cluster. 2. Add my hosts (I only have 2 so that is quick and easy). 3. Add a throw away data domain (This is needed to get the DC up so I can import the existing data domain). 4. Import (NOT new, import) the existing data domain. 5. Do to Storage->Storage Domains->VM import and import the VMs I want. 6. Same for templates and disks if needed. 7. After you have imported the VMs/Templates/Disks you can detach and remove the throw away data domain and the one you imported becomes the master domain. Note if you want to move VMs between your play ground and more serious system you can simply detach your data domain from the play ground, then attach it to the serious engine (so you have 2 engines, one play ground and one serious) and import which VMs you want. That way you won't run into issues with configuring networks and stuff like you experienced.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--ujRdKtglIhPz8reG7slXQyfJBb8Dum0Kz
Content-Type: multipart/mixed; boundary="kTkpiHIB81lTMdBrSQDkZk7ByBvTldWHY";
protected-headers="v1"
From: ~Stack~ <i.am.stack@gmail.com>
To: users@ovirt.org
Message-ID: <afd965b8-7a09-9fa2-ebed-a267f5487760@gmail.com>
Subject: Re: [ovirt-users] I broke my ovirt real good....
References: <816729cd-1ec6-f2e6-6986-2a18e83ab650@gmail.com>
<2691691.Lu8XbsuI1x@awels>
In-Reply-To: <2691691.Lu8XbsuI1x@awels>
--kTkpiHIB81lTMdBrSQDkZk7ByBvTldWHY
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
On 04/13/2018 07:16 AM, Alexander Wels wrote:
> On Thursday, April 12, 2018 6:26:07 PM EDT ~Stack~ wrote:
>> Greetings,
>>
>> So I did a over-confident-admin-makes-rookie-mistake. I changed a bunc=
h
>> of things all back-to-back and thus don't actually know what broke. :-=
D
>>
>> The only two real "big" changes were:
>> * Upgrade from 4.2.1 to 4.2.2
>> * change my ovirtmgmt network
>>
>> The update I followed the upgrade procedures and I thought it all went=
>> pretty well. Because I am moving it from a testing into what I hope wi=
ll
>> be a more heavily used environment, I changed my ovirtmgmt network fro=
m
>> 192.168.100.0/24 to 192.168.101.0/24 via the web-gui.
>>
>> That was a touch tricker than just a change as I had to poke the
>> management engine host to be reachable on both network for a while, th=
en
>> it just seemed OK.
>>
>> What's happening is:
>> * I can no longer migrate a vm from one host to the other.
>> * If I try to do a "reinstall" it dies.
>> * There is some serious network lag between my hosts on a 10Gb network=
=2E
>> * I've got all kinds of python2.4 failures in my vdsm and mom logs.
>>
>> Those are least the biggies.
>>
>> So while I was planning on moving this to a more active use case, righ=
t
>> now - it is all still my play ground. I would REALLY hate to lose the
>> VM's but everything else can go and be rebuilt.
>>
>> Given that I've somehow really broke this system pretty good, would it=
>> be more advisable to blow away and rebuild it ALL or can I simply dele=
te
>> the hypervisor hosts and rebuild them?
>>
>> Thoughts?
>>
>> Thanks!
>> ~Stack~
>=20
> As long as you don't destroy the data on your data domain you can rebui=
ld the=20
> engine and hosts and then import the existing data domain without too m=
any=20
> issues. I have destroyed my engine database many times, and I am still =
using=20
> the same VMs from the same data domain.
>=20
> Here is what I do when I mess up my database to the point I have to mak=
e a new=20
> one:
>=20
> 1. Recreate the engine and database, so that I have basically have an e=
mpty=20
> engine with no hosts and VMs.
> 1.1 (Optional) make a new DC that is not default. and add a cluster.
> 2. Add my hosts (I only have 2 so that is quick and easy).
> 3. Add a throw away data domain (This is needed to get the DC up so I c=
an=20
> import the existing data domain).
> 4. Import (NOT new, import) the existing data domain.
> 5. Do to Storage->Storage Domains->VM import and import the VMs I want.=
> 6. Same for templates and disks if needed.
> 7. After you have imported the VMs/Templates/Disks you can detach and r=
emove=20
> the throw away data domain and the one you imported becomes the master =
domain.
>=20
> Note if you want to move VMs between your play ground and more serious =
system=20
> you can simply detach your data domain from the play ground, then attac=
h it to=20
> the serious engine (so you have 2 engines, one play ground and one seri=
ous)=20
> and import which VMs you want. That way you won't run into issues with =
> configuring networks and stuff like you experienced.
>=20
Thanks for that help. I did that and everything looks fantastic...except
I can't migrate VM's. :-/
It just sits there and in the log files there is the below messages
repeating. It's like it doesn't care for the fact that this was an
imported domain or something.
Thoughts?
Thanks!
~Stack~
2018-04-13 16:58:59,920-0500 ERROR (monitor/232975a) [storage.Monitor]
Setting up monitor for 232975ad-1771-4b6b-afda-958f7b745867 failed
(monitor:329)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
326, in _setupLoop
self._setupMonitor()
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
348, in _setupMonitor
self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 158, in
wrapper
value =3D meth(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
366, in _produceDomain
self.domain =3D sdCache.produce(self.sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110,
in produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51,
in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134,
in _realProduce
domain =3D self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151,
in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176,
in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'232975ad-1771-4b6b-afda-958f7b745867',)
2018-04-13 16:58:59,923-0500 ERROR (monitor/bc975a4) [storage.Monitor]
Setting up monitor for bc975a4c-6c38-4248-b3f7-a26945f23693 failed
(monitor:329)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
326, in _setupLoop
self._setupMonitor()
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
348, in _setupMonitor
self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 158, in
wrapper
value =3D meth(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
366, in _produceDomain
self.domain =3D sdCache.produce(self.sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110,
in produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51,
in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134,
in _realProduce
domain =3D self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151,
in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176,
in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'bc975a4c-6c38-4248-b3f7-a26945f23693',)
--kTkpiHIB81lTMdBrSQDkZk7ByBvTldWHY--
--ujRdKtglIhPz8reG7slXQyfJBb8Dum0Kz
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJa0TO/AAoJELkej+ysXJPmUBsP/19UzsPB/qYad0aF9uBA9US3
9MAaHPAUx4IWUFViHGxN/B5BRxXqg0lnYsC2EEGNcKOlz9n66zVTyoegV5P9ZMSZ
iz206e3RwtlA/y2TV/E3HhuZnD7Z+ImTWZLmloKagacxdLMJ0RrUzoo4uhOdyA6J
xHs1XE/RlvqgSkb55zsMz17a06S/LIBNrGn/BB/7jIlHTC/gEcojOmVtYGpPOBsw
NJWtEdqYXx6wHs77P3n6Yi9toDEGSWWAHT+h95dMF0wSBo2Jw1OJmnbmzUJTQtp1
MQtojJkvv2kbBc9NWonOeBNYxNzZyy+jCV+bk6PfobJ2Gb8he8RhVuksALJb+bfI
U/e4frxfu3Gpm9XpGD3TJBAkIXOM+rwJ6TOjfD79KAalO42BtV7l+FQaIS/YQwWZ
/ta6GkP3UAuG2SYz5Hf9/l8kr3Odvun56dwlWPlw6DMKTeUGuP/wZPK7gPAXCpYS
cnH1PpNE0ctqzGB889XtSUBsI+lmr4n23ysG/IgvY3HBlTa+wmaLnDugQCwtqdHt
rW00L4HK62APctym2BysS3MRU7KzTG4Zkh2CzkaLsZvva0HkWYKDHfJ4EcUL1wiW
MYvmFNN83eNDXwvVrduCqRjZPLCiEH9jc0IdjgTBq+yUiX4tsMnfLV44wwnfHxhD
tTqbcSt24kE1GNlqc9aw
=eh58
-----END PGP SIGNATURE-----
--ujRdKtglIhPz8reG7slXQyfJBb8Dum0Kz--
On Friday, April 13, 2018 6:48:31 PM EDT ~Stack~ wrote:
Don't know too much about the VDSM side of things. But obviously its looking for a storage domain it can't find anymore. You can try restarting VDSM (won't affect running VMs) and see if rescans the available storage domains and won't try to access it during the migration of the VMs. Other than that I don't know.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --1T2PnHwioHDz98CXKaytGd6p5iLXElj7K Content-Type: multipart/mixed; boundary="5Ewoxldw33XWpL7fb6eCu6BGwekvvyavW"; protected-headers="v1" From: ~Stack~ <i.am.stack@gmail.com> To: users@ovirt.org Message-ID: <362d0961-23c4-dc36-e45a-726d4e065bfb@gmail.com> Subject: Re: [ovirt-users] I broke my ovirt real good.... References: <816729cd-1ec6-f2e6-6986-2a18e83ab650@gmail.com> <2691691.Lu8XbsuI1x@awels> <afd965b8-7a09-9fa2-ebed-a267f5487760@gmail.com> <7662271.cb9bfaddkY@awels> In-Reply-To: <7662271.cb9bfaddkY@awels> --5Ewoxldw33XWpL7fb6eCu6BGwekvvyavW Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 04/16/2018 10:02 AM, Alexander Wels wrote:
No worries. Thanks for responding. One of my hosts has gone berserk anyway so I'm just going to do a complete fresh reinstall tomorrow. The host says "Host has no default route" which is a load of bull. There's nothing wrong with the default route or network connectivity. However, oVirt puts it into non-opperational where it will sit for about 20 minutes. When it finally actually stops that process, it immediately (milliseconds later) puts it into "activating" but then complain about the default route and the whole process starts over again. There's something wrong with this install so I'm just going to take the nuke-it-from-orbit-and-start-over approach tomorrow morning. ~Stack~ --5Ewoxldw33XWpL7fb6eCu6BGwekvvyavW-- --1T2PnHwioHDz98CXKaytGd6p5iLXElj7K Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJa1R6vAAoJELkej+ysXJPmOjEP/RfPwi/O+A7XJiGF4biF3ECs pHOfpZFajW6jHOO1TBVJmAM4Wii9//UCZj4Ds/kroe9pfqOuR4ABlapcOl8U6/Xo LsdoyTVW26jU80V7oGy9aVzQOEcnJ/1kQ4o+zPEeOdyP3gITmKvlsTPKOtdYREQW vIHBJnfgjaduLtrauDsGGrhuwOV+HOP7UqE9vlgMPoucCMKXPUxp8FI6poo560bV R7NhkvaJCfKax2iWio0S7WpJkb4Im6FDrf6jZPpWCI+UCHxDMnh66DZU2ApiVXzI ThNB8GXapAL4UcKUZKhGASFHQkt+m6NW2jKak9jk3vUzngU5TfitHAPTnmVjYi/3 g4aNLzWk7FmTRO0RhwdCV48hCUctLvrZyQcMSVloqYQZBIA1dYteAFdAdsllq/Uy BkaU5ElngHtsT6ltubDPCfQWgaC5LaWG2emWEl/BL5KAWS6X6lm4nnDf1PKTgICu 9s2ueenRI4hB+AA42ZKGzO96Ahrcsw/5zmimOKR6krmAnxXIhKAs9TmEG+bG83js o2ZmJnT/Zrw61IeFnkY8QDHXciahkP0IAhcGtIPyZZFjlVPrMhduOopLMaalnhu0 liANxadsqeuY4rmpRIxTjjiufEXnRZXWRNG/oQ1d5vjswLbs4Fas66oDN9KoeHST ZlmEkgF0URDmpTBIFHHU =fGno -----END PGP SIGNATURE----- --1T2PnHwioHDz98CXKaytGd6p5iLXElj7K--
participants (3)
-
Alexander Wels -
Michael Mortensen (MCMR) -
~Stack~