On 31 Mar 2017, at 15:49, Nelson Lameiras = <nelson.lameiras@lyra-network.com> wrote: =20 Hello, =20 We had a rather unpleseant surprise while upgrading our production = datacenters to oVirt 4.0 =3D> 4.1 (although I think this problem is not = related to oVirt 4.1).=20 Some critical VMs simply refused to reboot without any particular = error message other than "VM failed to boot" =20 After a stressfull 2h of investigation, it turns out that the =
Their "custom compatilibity setting" was set to 3.6 !! When changing =
=20 Neverthess, there is little to none information/documentation about =
--Apple-Mail=_D1AB200B-A950-4A9B-B743-11B90EA156F0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 problematic VMs shared the same "exotic" condition. They were not = rebooted since the latest major upgrade 3.6 =3D> 4.0 (I know, this is = not a good practise and we underestimated it's priority, we will change = our reboot policies) it=E2=80=99s indeed not a good practice. You can=E2=80=99t use the = features (those related to hw) without updating the guest HW and that = will only happen once you shut the VM down. If you can=E2=80=99t do that = then just keep the old cluster level, that is fully supported. this setting to "Empty", The VM booted normaly. Hard to find, easy to = fix !! ;) the usefullness of this setting, and I have a hard time understanding = the consequences of changing it manually.
=20 Our oVirt datacenter (which has engine 4.1.1, but some hosts on 4.0, = so it's still on 4.0 compatibility) hosts currently +- 200 Vms, - 10 with "custom compatilibity setting" set to 3.6 - which will not = (reb)boot in current state, - 30some with "custom compatilibity setting" set to 4.0 - which = (re)boot normally=20 - the rest with "custom compatilibity setting" set to "empty" - which = (re)boot normally=20 =20 So can a oVirt guru please answer the following questions : =20 - What does this setting do? which are the consequences of changing it = manually ?
it emulates the corresponding guest hardware and engine feature = behavior.
- Is it "normal" that a VM not reboooted since the 3.6 update does not = boot if a new major upgrade is done on hostig host ? (maybe a exotic bug = worth correcting ?)
no, it=E2=80=99s a bug = https://bugzilla.redhat.com/show_bug.cgi?id=3D1436325 = <https://bugzilla.redhat.com/show_bug.cgi?id=3D1436325> will be fixed in 4.1.2
- Which are the possible consequences of manually setting this setting = to "Empty" in all my VMs (running or stopped) ?
then it will use the cluster settings. that is the =E2=80=9Cnormal=E2=80=9D= state
- Which events will change this setting automatically ? (cluster major = version upgrade, first reboot after upgrade, ...) ?
cluster level upgrade while VM is running. Since the VM is running the = changes to hardware cannot be applied and it is temporarily reconfigured = to use the previous cluster level. On VM shutdown the settings are all = updated and the field set to empty. there was another bug (hopefully fixed) which kept the value there even = on shutdown. Just edit and put =E2=80=9Cempty=E2=80=9D, then it will = start as a regular VM in its cluster.
- Some of my VMs have custom_compatibility_version set to 4.0 (in REST = API) eventough they have been recently rebooted and "custom = compatilibity setting" is empty in GUI, who's this possible ?
was it done before the reboot? are there pending changes to be applied = perhaps? Thanks, michal
=20 cordialement, regards, =20 <element-signature_logo_lyra_115x94.jpg> <https://www.lyra-network.com/> Nelson LAMEIRAS Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks = engineer Tel: +33 5 32 09 09 70 nelson.lameiras@lyra-network.com = <mailto:nelson.lameiras@lyra-network.com> www.lyra-network.com <https://www.lyra-network.com/> | www.payzen.eu = <https://payzen.eu/> <element-signature_logo_YouTube_32x28.jpg> = <https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA> <element-signature_logo_LinkedIn_41x28.jpg> = <https://www.linkedin.com/company/lyra-network_2> <element-signature_logo_Twitter_42x28.jpg> = <https://twitter.com/LyraNetwork> <element-signature_payzen_61x28.jpg> <https://payzen.eu/> Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE =20 _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_D1AB200B-A950-4A9B-B743-11B90EA156F0 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 31 Mar 2017, at 15:49, Nelson Lameiras <<a = href=3D"mailto:nelson.lameiras@lyra-network.com" = class=3D"">nelson.lameiras@lyra-network.com</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div class=3D""><div = style=3D"font-family: arial, helvetica, sans-serif; font-size: 12pt;" = class=3D""><div class=3D"">Hello,<br class=3D""></div><div class=3D""><br = data-mce-bogus=3D"1" class=3D""></div><div class=3D"">We had a rather = unpleseant surprise while upgrading our production datacenters to oVirt = 4.0 =3D> 4.1 (although I think this problem is not related to oVirt = 4.1). </div><div class=3D"">Some critical VMs simply refused to reboot = without any particular error message other than "VM failed to boot"<br = data-mce-bogus=3D"1" class=3D""></div><div class=3D""><br = data-mce-bogus=3D"1" class=3D""></div><div class=3D"">After a stressfull = 2h of investigation, it turns out that the problematic VMs shared the = same "exotic" condition. They were not rebooted since the latest major = upgrade 3.6 =3D> 4.0 (I know, this is not a good practise and we = underestimated it's priority, we will change our reboot policies)<br = data-mce-bogus=3D"1" = class=3D""></div></div></div></div></blockquote><div><br = class=3D""></div>it=E2=80=99s indeed not a good practice. You can=E2=80=99= t use the features (those related to hw) without updating the guest HW = and that will only happen once you shut the VM down. If you can=E2=80=99t = do that then just keep the old cluster level, that is fully = supported.</div><div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div class=3D""><div style=3D"font-family: = arial, helvetica, sans-serif; font-size: 12pt;" class=3D""><div = class=3D"">Their "custom compatilibity setting" was set to 3.6 !! When = changing this setting to "Empty", The VM booted normaly. Hard to find, = easy to fix !! ;)<br data-mce-bogus=3D"1" class=3D""></div><div = class=3D""><br data-mce-bogus=3D"1" class=3D""></div><div = class=3D"">Neverthess, there is little to none information/documentation = about the usefullness of this setting, and I have a hard time = understanding the consequences of changing it manually.<br = data-mce-bogus=3D"1" class=3D""></div><div class=3D""><br = data-mce-bogus=3D"1" class=3D""></div><div class=3D"">Our oVirt = datacenter (which has engine 4.1.1, but some hosts on 4.0, so it's = still on 4.0 compatibility) hosts currently +- 200 Vms, </div><div = class=3D"">- 10 with "custom compatilibity setting" set to 3.6 - which = will not (reb)boot in current state, </div><div class=3D"">- 30some with = "custom compatilibity setting" set to 4.0 - which (re)boot normally <br = data-mce-bogus=3D"1" class=3D""></div><div class=3D"">- the rest with = "custom compatilibity setting" set to "empty" - which (re)boot = normally <br data-mce-bogus=3D"1" class=3D""></div><div = class=3D""><br data-mce-bogus=3D"1" class=3D""></div><div class=3D"">So = can a oVirt guru please answer the following questions :<br = data-mce-bogus=3D"1" class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D"">- What does this setting do? which are = the consequences of changing it manually ?<br data-mce-bogus=3D"1" = class=3D""></div></div></div></div></blockquote><div><br = class=3D""></div>it emulates the corresponding guest hardware and engine = feature behavior.</div><div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div class=3D""><div style=3D"font-family: = arial, helvetica, sans-serif; font-size: 12pt;" class=3D""><div = class=3D"">- Is it "normal" that a VM not reboooted since the 3.6 update = does not boot if a new major upgrade is done on hostig host ? (maybe a = exotic bug worth correcting ?)<br data-mce-bogus=3D"1" = class=3D""></div></div></div></div></blockquote><div><br = class=3D""></div>no, it=E2=80=99s a bug <a = href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1436325" = class=3D"">https://bugzilla.redhat.com/show_bug.cgi?id=3D1436325</a></div>= <div>will be fixed in 4.1.2</div><div><br = class=3D""></div><div><blockquote type=3D"cite" class=3D""><div = class=3D""><div class=3D""><div style=3D"font-family: arial, helvetica, = sans-serif; font-size: 12pt;" class=3D""><div class=3D"">- Which are the = possible consequences of manually setting this setting to "Empty" in all = my VMs (running or stopped) ?<br data-mce-bogus=3D"1" = class=3D""></div></div></div></div></blockquote><div><br = class=3D""></div>then it will use the cluster settings. that is the = =E2=80=9Cnormal=E2=80=9D state</div><div><br class=3D""><blockquote = type=3D"cite" class=3D""><div class=3D""><div class=3D""><div = style=3D"font-family: arial, helvetica, sans-serif; font-size: 12pt;" = class=3D""><div class=3D"">- Which events will change this setting = automatically ? (cluster major version upgrade, first reboot after = upgrade, ...) ?<br data-mce-bogus=3D"1" = class=3D""></div></div></div></div></blockquote><div><br = class=3D""></div>cluster level upgrade while VM is running. Since the VM = is running the changes to hardware cannot be applied and it is = temporarily reconfigured to use the previous cluster level. On VM = shutdown the settings are all updated and the field set to = empty.</div><div><div>there was another bug (hopefully fixed) which kept = the value there even on shutdown. Just edit and put =E2=80=9Cempty=E2=80=9D= , then it will start as a regular VM in its cluster.</div><div = class=3D""><br class=3D""></div><blockquote type=3D"cite" class=3D""><div = class=3D""><div class=3D""><div style=3D"font-family: arial, helvetica, = sans-serif; font-size: 12pt;" class=3D""><div class=3D"">- Some of my = VMs have custom_compatibility_version set to 4.0 (in REST = API) eventough they have been recently rebooted and "custom = compatilibity setting" is empty in GUI, who's this possible ?<br = data-mce-bogus=3D"1" = class=3D""></div></div></div></div></blockquote><div><br = class=3D""></div>was it done before the reboot? are there pending = changes to be applied perhaps?</div><div><br = class=3D""></div><div>Thanks,</div><div>michal</div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = class=3D""><div style=3D"font-family: arial, helvetica, sans-serif; = font-size: 12pt;" class=3D""><div class=3D""><br data-mce-bogus=3D"1" = class=3D""></div><div data-marker=3D"__SIG_PRE__" class=3D""><div = class=3D"">cordialement, regards,</div><div class=3D""><br = data-mce-bogus=3D"1" class=3D""></div><table style=3D"margin: 0px; = table-layout: fixed; border: none; width: 345pt; padding: 0px;" = data-mce-style=3D"margin: 0px; table-layout: fixed; border: none; width: = 345pt; padding: 0px;" width=3D"460" cellspacing=3D"0" cellpadding=3D"0" = border=3D"0" class=3D""><tbody class=3D""><tr class=3D""><td = style=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; = color: #2b3c7a; line-height: 16px; width: 86.25pt; text-align: left;" = data-mce-style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 86.25pt; = text-align: left;" width=3D"115" valign=3D"top" class=3D""><a = href=3D"https://www.lyra-network.com/" style=3D"border: none; = text-decoration: none; width: 100%; display: block; clear: both;" = data-mce-href=3D"https://www.lyra-network.com/" data-mce-style=3D"border: = none; text-decoration: none; width: 100%; display: block; clear: both;" = class=3D""><span = id=3D"cid:1504d440a45e25d470766a589e361163b925cd4d@zimbra"><element-sig= nature_logo_lyra_115x94.jpg></span></a><br data-mce-bogus=3D"1" = class=3D""></td><td style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 258.75pt;" = data-mce-style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 258.75pt;" = width=3D"345" valign=3D"top" class=3D""><table style=3D"margin: 0; = width: 258.75pt; table-layout: fixed; border: none;" = data-mce-style=3D"margin: 0; width: 258.75pt; table-layout: fixed; = border: none;" width=3D"345" cellspacing=3D"0" cellpadding=3D"0" = border=3D"0" class=3D""><tbody class=3D""><tr class=3D""><td = style=3D"font-size: 16px; font-family: Arial,sans-serif; color: #2b3c7a; = line-height: 18px; font-weight: bold;" data-mce-style=3D"font-size: = 16px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 18px; = font-weight: bold;" class=3D"">Nelson LAMEIRAS</td></tr><tr class=3D""><td= style=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-size: = 11px; line-height: 16px;" data-mce-style=3D"font-family: = Arial,sans-serif; color: #2b3c7a; font-size: 11px; line-height: 16px;" = class=3D"">Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux<span = style=3D"font-family: Arial,sans-serif; color: #337ab7; font-size: = 11px;" data-mce-style=3D"font-family: Arial,sans-serif; color: #337ab7; = font-size: 11px;" class=3D""> / Systems and Networks = engineer</span></td></tr><tr class=3D""><td style=3D"line-height: 16px; = font-family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" = data-mce-style=3D"line-height: 16px; font-family: 'Arial',sans-serif; = color: #2b3c7a; font-size: 11px;" class=3D""><span style=3D"font-family: = Arial,sans-serif; color: #2b3c7a; font-size: 11px;" = data-mce-style=3D"font-family: Arial,sans-serif; color: #2b3c7a; = font-size: 11px;" class=3D"">Tel: +33 5 32 09 09 70</span></td></tr><tr = class=3D""><td style=3D"line-height: 16px; font-family: = 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" = data-mce-style=3D"line-height: 16px; font-family: 'Arial',sans-serif; = color: #2b3c7a; font-size: 11px;" class=3D""><a = href=3D"mailto:nelson.lameiras@lyra-network.com" style=3D"font-size: = 11px; font-family: Arial,sans-serif; line-height: 16px; text-decoration: = none;" data-mce-href=3D"mailto:nelson.lameiras@lyra-network.com" = data-mce-style=3D"font-size: 11px; font-family: Arial,sans-serif; = line-height: 16px; text-decoration: none;" = class=3D"">nelson.lameiras@lyra-network.com</a><br data-mce-bogus=3D"1" = class=3D""></td></tr><tr class=3D""><td style=3D"line-height: 16px; = font-family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" = data-mce-style=3D"line-height: 16px; font-family: 'Arial',sans-serif; = color: #2b3c7a; font-size: 11px;" class=3D""><a = href=3D"https://www.lyra-network.com/" style=3D"font-size: 11px; = font-family: Arial,sans-serif; line-height: 16px; border: none; = text-decoration: none;" data-mce-href=3D"https://www.lyra-network.com/" = data-mce-style=3D"font-size: 11px; font-family: Arial,sans-serif; = line-height: 16px; border: none; text-decoration: none;" = class=3D"">www.lyra-network.com</a> <a href=3D"https://payzen.eu/" = style=3D"font-size: 11px; font-family: Arial,sans-serif; line-height: = 16px; border: none; text-decoration: none;" = data-mce-href=3D"https://payzen.eu" data-mce-style=3D"font-size: 11px; = font-family: Arial,sans-serif; line-height: 16px; border: none; = text-decoration: none;" class=3D""> | www.payzen.eu</a><br = data-mce-bogus=3D"1" = class=3D""></td></tr></tbody></table></td></tr></tbody></table><table = style=3D"margin: 0; table-layout: fixed; border: none; width: 345pt;" = data-mce-style=3D"margin: 0; table-layout: fixed; border: none; width: = 345pt;" width=3D"460" cellspacing=3D"0" cellpadding=3D"0" = class=3D""><tbody class=3D""><tr class=3D""><td style=3D"margin: 0; = font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; = line-height: 16px; width: 24pt; text-align: left;" = data-mce-style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 24pt; = text-align: left;" width=3D"32" valign=3D"top" class=3D""><a = href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" = style=3D"border: none; text-decoration: none; margin: 0; float: left; = display: inline-block; width: 32px;" = data-mce-href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA"= data-mce-style=3D"border: none; text-decoration: none; margin: 0; = float: left; display: inline-block; width: 32px;" class=3D""><span = id=3D"cid:089c696a3ee09447fa2f5d20d0074bf34dc05be8@zimbra"><element-sig= nature_logo_YouTube_32x28.jpg></span></a><br data-mce-bogus=3D"1" = class=3D""></td><td style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; = text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; = font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: = 30.75pt; text-align: left;" width=3D"41" valign=3D"top" class=3D""><a = href=3D"https://www.linkedin.com/company/lyra-network_2" style=3D"border: = none; text-decoration: none; margin: 0; float: left; display: = inline-block; width: 41px;" = data-mce-href=3D"https://www.linkedin.com/company/lyra-network_2" = data-mce-style=3D"border: none; text-decoration: none; margin: 0; float: = left; display: inline-block; width: 41px;" class=3D""><span = id=3D"cid:11845310ca84fd4e4def524810120e980ff072e5@zimbra"><element-sig= nature_logo_LinkedIn_41x28.jpg></span></a><br data-mce-bogus=3D"1" = class=3D""></td><td style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 31.5pt; = text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; = font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: = 31.5pt; text-align: left;" width=3D"42" valign=3D"top" class=3D""><a = href=3D"https://twitter.com/LyraNetwork" style=3D"border: none; = text-decoration: none; margin: 0; float: left; display: inline-block; = width: 42px;" data-mce-href=3D"https://twitter.com/LyraNetwork" = data-mce-style=3D"border: none; text-decoration: none; margin: 0; float: = left; display: inline-block; width: 42px;" class=3D""><span = id=3D"cid:c2735661868f46ef1f20b767bf188c91be0f52bd@zimbra"><element-sig= nature_logo_Twitter_42x28.jpg></span></a><br data-mce-bogus=3D"1" = class=3D""></td><td style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 258.75pt; = text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; = font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: = 258.75pt; text-align: left;" width=3D"345" valign=3D"top" class=3D""><a = href=3D"https://payzen.eu/" style=3D"border: none; text-decoration: = none; margin: 0; float: left; display: inline-block; width: 61px;" = data-mce-href=3D"https://payzen.eu" data-mce-style=3D"border: none; = text-decoration: none; margin: 0; float: left; display: inline-block; = width: 61px;" class=3D""><span = id=3D"cid:4003b71aaa9799afe7266b76ecec0006c92f65bb@zimbra"><element-sig= nature_payzen_61x28.jpg></span></a><br data-mce-bogus=3D"1" = class=3D""></td></tr></tbody></table><table style=3D"margin: 0; = table-layout: fixed; border: none; width: 345pt;" = data-mce-style=3D"margin: 0; table-layout: fixed; border: none; width: = 345pt;" width=3D"460" cellspacing=3D"0" cellpadding=3D"0" = class=3D""><tbody class=3D""><tr class=3D""><td class=3D""><hr = style=3D"background: #293b79 none repeat scroll 0% 0%; width: 100%; = height: 4px; color: #293b79; margin: 0px;" data-mce-style=3D"background: = #293b79 none repeat scroll 0% 0%; width: 100%; height: 4px; color: = #293b79; margin: 0px;" class=3D""></td></tr></tbody></table><table = style=3D"margin: 0; table-layout: fixed; border: none; width: 345pt;" = data-mce-style=3D"margin: 0; table-layout: fixed; border: none; width: = 345pt;" width=3D"460" cellspacing=3D"0" cellpadding=3D"0" = class=3D""><tbody class=3D""><tr class=3D""><td style=3D"margin: 0; = font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; = line-height: 16px; width: 30.75pt; text-align: left;" = data-mce-style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; = text-align: left;" class=3D""><div class=3D"">Lyra Network, 109 rue de = l'innovation, 31670 Lab=C3=A8ge, = FRANCE</div></td></tr></tbody></table><div class=3D""><br = class=3D""></div></div></div></div>_______________________________________= ________<br class=3D"">Users mailing list<br class=3D""><a = href=3D"mailto:Users@ovirt.org" class=3D"">Users@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_D1AB200B-A950-4A9B-B743-11B90EA156F0--