This is a multi-part message in MIME format. --------------31CF192FA29B9CBBDE0FE9E5 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit On 09/18/2017 11:12 AM, Simone Tiraboschi wrote:
On Mon, Sep 18, 2017 at 12:09 PM, Simone Tiraboschi <stirabos@redhat.com <mailto:stirabos@redhat.com>> wrote:
On Sun, Sep 17, 2017 at 11:14 AM, Eyal Edri <eedri@redhat.com <mailto:eedri@redhat.com>> wrote:
On Sun, Sep 17, 2017 at 11:50 AM, Yaniv Kaul <ykaul@redhat.com <mailto:ykaul@redhat.com>> wrote:
On Sun, Sep 17, 2017 at 11:47 AM, Eyal Edri <eedri@redhat.com <mailto:eedri@redhat.com>> wrote:
Hi,
It looks like HE suite ( both 'master' and '4.1' ) is failing constantly, most likely due to 7.4 updates.
I'm investigating the issue on master. In my case I choose to configure the engine VM with a static IP address and engine-setup failed on the engine VM since it wasn't able to check available OVN related packages.
So we have two distinct issues here: 1. we are executing engine-setup with --offline cli option but the OVN plugins are ignoring it.
2. the engine VM has no connectivity. I dig it a bit and I found that the default gateway wasn't configured on the engine VM although it's correctly set in cloud-init meta-data file. So it seams that on 7.4 cloud-init is failing to set the default gateway:
[root@enginevm ~]# nmcli con show "System eth0" | grep -i GATEWAY connection.gateway-ping-timeout: 0 ipv4.gateway: -- ipv6.gateway: -- IP4.GATEWAY: -- IP6.GATEWAY: fe80::c4ee:3eff:fed5:fad9 [root@enginevm ~]# nmcli con modify "System eth0" ipv4.gateway Error: value for 'ipv4.gateway' is missing. [root@enginevm ~]# [root@enginevm ~]# nmcli con show "System eth0" | grep -i GATEWAY connection.gateway-ping-timeout: 0 ipv4.gateway: -- ipv6.gateway: -- IP4.GATEWAY: -- IP6.GATEWAY: fe80::c4ee:3eff:fed5:fad9 [root@enginevm ~]# nmcli con modify "System eth0" ipv4.gateway 192.168.1.1 [root@enginevm ~]# nmcli con reload "System eth0" [root@enginevm ~]# nmcli con up "System eth0" Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3) [root@enginevm ~]# nmcli con show "System eth0" | grep -i GATEWAY connection.gateway-ping-timeout: 0 ipv4.gateway: 192.168.1.1 ipv6.gateway: -- IP4.GATEWAY: 192.168.1.1 IP6.GATEWAY: fe80::c4ee:3eff:fed5:fad9 [root@enginevm ~]# mount /dev/sr0 /mnt/ mount: /dev/sr0 is write-protected, mounting read-only [root@enginevm ~]# cat /mnt/meta-data instance-id: d8b22f43-1565-44e2-916f-f211c7e07f13 local-hostname: enginevm.localdomain network-interfaces: | auto eth0 iface eth0 inet static address 192.168.1.204 network 192.168.1.0 netmask 255.255.255.0 broadcast 192.168.1.255 gateway 192.168.1.1
An upstream user was also reporting that he updated his host and his engine VM to Centos 7.4 and it failed to reboot the engine VM with 7.4 kernel hanging at "Probing EDD (edd=off to disable)...ok". He manually forced the old 7.3 kernel via grub menu and his engine VM correctly booted. I wasn't able to reproduce it here.
This is a firmware issue for enhanced disk drive. to work around it we can add linux edd=off to the kernel boot parameters.
So there is no suspected patch from oVirt side that might have caused it.
It's the firewall. I've fixed it[1] and specifically[2] but probably not completely.
Great! Wasn't aware your patch address that, I've replied on the patch itself, but I think we need to split the fix to 2 seperate patches.
Perhaps we should try to take[2] separately. Y.
[1] https://gerrit.ovirt.org/#/c/81766/ <https://gerrit.ovirt.org/#/c/81766/> [2] https://gerrit.ovirt.org/#/c/81766/3/common/deploy-scripts/setup_storage_uni... <https://gerrit.ovirt.org/#/c/81766/3/common/deploy-scripts/setup_storage_unified_he_extra_el7.sh>
It is probably also the reason why HC suites are failing, since they are using also HE for deployments.
I think this issue should BLOCK the Alpha release tomorrow, or at the minimum, we need to verify its an OST issue and not a real regression.
Links to relevant failures: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/37/con... <http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/37/consoleFull> http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.1/33/consol... <http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.1/33/console>
Error snippet:
03:01:38 03:01:38 --== STORAGE CONFIGURATION ==-- 03:01:38 03:02:47 [ ERROR ] Error while mounting specified storage path: mount.nfs: No route to host 03:02:58 [WARNING] Cannot unmount /tmp/tmp2gkFwJ 03:02:58 [ ERROR ] Failed to execute stage 'Environment customization': mount.nfs: No route to host
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018 <tel:+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel <http://lists.ovirt.org/mailman/listinfo/devel>
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018 <tel:+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--------------31CF192FA29B9CBBDE0FE9E5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 09/18/2017 11:12 AM, Simone Tiraboschi wrote:<br> </div> <blockquote type="cite" cite="mid:CAN8-ONq8JmhpkDiODnJcNL6j7E3aFXYJ+bCT590K_VmzFU-LAA@mail.gmail.com"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote">On Mon, Sep 18, 2017 at 12:09 PM, Simone Tiraboschi <span dir="ltr"><<a href="mailto:stirabos@redhat.com" target="_blank" moz-do-not-send="true">stirabos@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote"><span class="gmail-">On Sun, Sep 17, 2017 at 11:14 AM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank" moz-do-not-send="true">eedri@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote"><span class="gmail-m_7974968019882020809gmail-">On Sun, Sep 17, 2017 at 11:50 AM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank" moz-do-not-send="true">ykaul@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote"><span>On Sun, Sep 17, 2017 at 11:47 AM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank" moz-do-not-send="true">eedri@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr">Hi, <div><br> </div> <div>It looks like HE suite ( both 'master' and '4.1' ) is failing constantly, most likely due to 7.4 updates.</div> </div> </blockquote> </span></div> </div> </div> </blockquote> </span></div> </div> </div> </blockquote> <div><br> </div> </span> <div>I'm investigating the issue on master.</div> <div>In my case I choose to configure the engine VM with a static IP address and engine-setup failed on the engine VM since it wasn't able to check available OVN related packages.</div> <div><br> </div> <div>So we have two distinct issues here:</div> <div>1. we are executing engine-setup with --offline cli option but the OVN plugins are ignoring it.</div> <div><br> </div> <div>2. the engine VM has no connectivity.</div> <div>I dig it a bit and I found that the default gateway wasn't configured on the engine VM although it's correctly set in cloud-init meta-data file.</div> <div>So it seams that on 7.4 cloud-init is failing to set the default gateway:</div> <div><br> </div> <div> <div>[root@enginevm ~]# nmcli con show "System eth0" | grep -i GATEWAY</div> <div>connection.gateway-ping-<wbr>timeout: 0</div> <div>ipv4.gateway: --</div> <div>ipv6.gateway: --</div> <div>IP4.GATEWAY: --</div> <div>IP6.GATEWAY: fe80::c4ee:3eff:fed5:fad9</div> <div>[root@enginevm ~]# nmcli con modify "System eth0" ipv4.gateway</div> <div>Error: value for 'ipv4.gateway' is missing.</div> <div>[root@enginevm ~]# </div> <div>[root@enginevm ~]# nmcli con show "System eth0" | grep -i GATEWAY</div> <div>connection.gateway-ping-<wbr>timeout: 0</div> <div>ipv4.gateway: --</div> <div>ipv6.gateway: --</div> <div>IP4.GATEWAY: --</div> <div>IP6.GATEWAY: fe80::c4ee:3eff:fed5:fad9</div> <div>[root@enginevm ~]# nmcli con modify "System eth0" ipv4.gateway 192.168.1.1</div> <div>[root@enginevm ~]# nmcli con reload "System eth0"</div> <div>[root@enginevm ~]# nmcli con up "System eth0"</div> <div>Connection successfully activated (D-Bus active path: /org/freedesktop/<wbr>NetworkManager/<wbr>ActiveConnection/3)</div> <div>[root@enginevm ~]# nmcli con show "System eth0" | grep -i GATEWAY</div> <div>connection.gateway-ping-<wbr>timeout: 0</div> <div>ipv4.gateway: 192.168.1.1</div> <div>ipv6.gateway: --</div> <div>IP4.GATEWAY: 192.168.1.1</div> <div>IP6.GATEWAY: fe80::c4ee:3eff:fed5:fad9</div> <div>[root@enginevm ~]# mount /dev/sr0 /mnt/</div> <div>mount: /dev/sr0 is write-protected, mounting read-only</div> <div>[root@enginevm ~]# cat /mnt/meta-data </div> <div>instance-id: d8b22f43-1565-44e2-916f-<wbr>f211c7e07f13</div> <div>local-hostname: enginevm.localdomain</div> <div>network-interfaces: |</div> <div> auto eth0</div> <div> iface eth0 inet static</div> <div> address 192.168.1.204</div> <div> network 192.168.1.0</div> <div> netmask 255.255.255.0</div> <div> broadcast 192.168.1.255</div> <div> gateway 192.168.1.1</div> </div> <div> <div class="gmail-h5"> <div><br> </div> </div> </div> </div> </div> </div> </blockquote> <div><br> </div> An upstream user was also reporting that he updated his host and his engine VM to Centos 7.4 and it failed to reboot the engine VM with 7.4 kernel hanging at<br> "Probing EDD (edd=off to disable)...ok". He manually forced the old 7.3 kernel via grub menu and his engine VM correctly booted.</div> <div class="gmail_quote">I wasn't able to reproduce it here.<br> </div> </div> </div> </blockquote> <br> <br> This is a firmware issue for enhanced disk drive. <br> to work around it we can add linux edd=off to the kernel boot parameters. <br> <blockquote type="cite" cite="mid:CAN8-ONq8JmhpkDiODnJcNL6j7E3aFXYJ+bCT590K_VmzFU-LAA@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div class="gmail-h5"> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"><span class="gmail-m_7974968019882020809gmail-"> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"><span> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div>So there is no suspected patch from oVirt side that might have caused it.</div> </div> </blockquote> <div><br> </div> </span> <div>It's the firewall. I've fixed it[1] and specifically[2] but probably not completely.</div> </div> </div> </div> </blockquote> <div><br> </div> </span> <div>Great! Wasn't aware your patch address that, I've replied on the patch itself, but I think we need to split the fix to 2 seperate patches.</div> <div> <div class="gmail-m_7974968019882020809gmail-h5"> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> <div>Perhaps we should try to take[2] separately.</div> <div>Y.</div> <div><br> </div> <div>[1] <a href="https://gerrit.ovirt.org/#/c/81766/" target="_blank" moz-do-not-send="true">https://gerrit.ovirt.org/#<wbr>/c/81766/</a></div> <div>[2] <a href="https://gerrit.ovirt.org/#/c/81766/3/common/deploy-scripts/setup_storage_uni..." target="_blank" moz-do-not-send="true">https://gerrit.ovirt.org/#<wbr>/c/81766/3/common/deploy-scrip<wbr>ts/setup_storage_unified_he_ex<wbr>tra_el7.sh</a></div> <div><br> </div> <div><br> </div> <div><br> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span> <div dir="ltr"> <div><br> </div> <div>It is probably also the reason why HC suites are failing, since they are using also HE for deployments.</div> <div><br> </div> <div>I think this issue should BLOCK the Alpha release tomorrow, or at the minimum, we need to verify its an OST issue and not a real regression.</div> <div><br> </div> <div>Links to relevant failures:</div> <div><a href="http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/37/con..." target="_blank" moz-do-not-send="true">http://jenkins.ovirt.org/job/o<wbr>virt-system-tests_he-basic-sui<wbr>te-master/37/consoleFull</a></div> <div><a href="http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.1/33/consol..." target="_blank" moz-do-not-send="true">http://jenkins.ovirt.org/job/o<wbr>virt-system-tests_he-basic-sui<wbr>te-4.1/33/console</a><br> </div> <div><br> </div> <div>Error snippet:</div> <div><br> </div> <div> <div>03:01:38 </div> <div>03:01:38 --== STORAGE CONFIGURATION ==--</div> <div>03:01:38 </div> <div>03:02:47 [ ERROR ] Error while mounting specified storage path: mount.nfs: No route to host</div> <div>03:02:58 [WARNING] Cannot unmount /tmp/tmp2gkFwJ</div> <div>03:02:58 [ ERROR ] Failed to execute stage 'Environment customization': mount.nfs: No route to host</div> </div> <div><br clear="all"> <div><br> </div> -- <br> <div class="gmail-m_7974968019882020809gmail-m_-7053372602613947594m_3604939481102820758gmail-m_6510779019963059883gmail_signature"> <div dir="ltr"> <div> <div dir="ltr"> <div> <div dir="ltr"> <div> <div dir="ltr"> <div> <p style="font-family:overpass,sans-serif;margin:0px;padding:0px;font-size:14px;text-transform:uppercase;font-weight:bold"><font color="#cc0000">Eyal edri</font></p> <p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><br> </p> <p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">ASSOCIATE MANAGER</p> <p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">RHV DevOps</p> <p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">EMEA VIRTUALIZATION R&D</p> <p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><br> </p> <p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank" moz-do-not-send="true">Red Hat EMEA</a></p> <table style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium" border="0"> <tbody> <tr> <td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank" moz-do-not-send="true"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-blac..." moz-do-not-send="true" width="90" height="auto"></a></td> <td style="font-size:10px"><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank" moz-do-not-send="true">TRIED. TESTED. TRUSTED.</a></td> </tr> </tbody> </table> </div> <div>phone: <a href="tel:+972%209-769-2018" value="+97297692018" target="_blank" moz-do-not-send="true">+972-9-7692018</a><br> irc: eedri (on #tlv #rhev-dev #rhev-integ)</div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <br> </span>______________________________<wbr>_________________<br> Devel mailing list<br> <a href="mailto:Devel@ovirt.org" target="_blank" moz-do-not-send="true">Devel@ovirt.org</a><br> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br> </blockquote> </div> <br> </div> </div> </blockquote> </div> </div> </div> <div> <div class="gmail-m_7974968019882020809gmail-h5"><br> <br clear="all"> <div><br> </div> -- <br> <div class="gmail-m_7974968019882020809gmail-m_-7053372602613947594gmail_signature"> <div dir="ltr"> <div> <div dir="ltr"> <div> <div dir="ltr"> <div> <div dir="ltr"> <div> <p style="font-family:overpass,sans-serif;margin:0px;padding:0px;font-size:14px;text-transform:uppercase;font-weight:bold"><font color="#cc0000">Eyal edri</font></p> <p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><br> </p> <p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">ASSOCIATE MANAGER</p> <p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">RHV DevOps</p> <p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">EMEA VIRTUALIZATION R&D</p> <p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><br> </p> <p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank" moz-do-not-send="true">Red Hat EMEA</a></p> <table style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium" border="0"> <tbody> <tr> <td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank" moz-do-not-send="true"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-blac..." moz-do-not-send="true" width="90" height="auto"></a></td> <td style="font-size:10px"><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank" moz-do-not-send="true">TRIED. TESTED. TRUSTED.</a></td> </tr> </tbody> </table> </div> <div>phone: <a href="tel:+972%209-769-2018" value="+97297692018" target="_blank" moz-do-not-send="true">+972-9-7692018</a><br> irc: eedri (on #tlv #rhev-dev #rhev-integ)</div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> </div> </div> <br> </div> </div> </blockquote> </div> <br> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Infra mailing list <a class="moz-txt-link-abbreviated" href="mailto:Infra@ovirt.org">Infra@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/infra">http://lists.ovirt.org/mailman/listinfo/infra</a> </pre> </blockquote> <p><br> </p> </body> </html> --------------31CF192FA29B9CBBDE0FE9E5--