[ovirt-users] Solved: Re: 3.5 to 3.6 upgrade stuck

[Simone Tiraboschi]

On Fri, Jul 22, 2016 at 9:58 AM, Simone Tiraboschi <stirabos at redhat.com> wrote:
> On Fri, Jul 22, 2016 at 4:11 AM, Robert Story <rstory at tislabs.com> wrote:
>> On Thu, 21 Jul 2016 16:04:41 -0400 Robert wrote:
>> RS> Thread-1::config::278::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
>> RS>      ::(refresh_local_conf_file) local conf file was correctly written
>> RS>
>> RS> And then .... nothing. It just hangs. Nothing more is logged Thread-1.
>>
>> So I started digging around the the python source, starting from
>> refresh_local_conf_file. I ended up in ./broker/notifications.py, in
>> send_email. I added some logging:
>>
>> def send_email(cfg, email_body):
>>     """Send email."""
>>
>>     logger = logging.getLogger("%s.Notifications" % __name__)
>>
>>     try:
>>         logger.debug("#### setting up smtp 1")
>>         server = smtplib.SMTP(cfg["smtp-server"], port=cfg["smtp-port"])
>>         logger.debug("#### setting up smtp 2")
>>         ...
>>
>> Now the final messages are:
>>
>> Thread-1::DEBUG::2016-07-21 21:35:05,280::config::278::
>>   ovirt_hosted_engine_ha.broker.notifications.Notifications.config::
>>   (refresh_local_conf_file) local conf file was correctly written
>> Thread-1::DEBUG::2016-07-21 21:35:05,282::notifications::27::
>>   ovirt_hosted_engine_ha.broker.notifications.Notifications::
>>   (send_email) #### setting up smtp 1
>>
>>
>> So the culprit is:
>>
>>         server = smtplib.SMTP(cfg["smtp-server"], port=cfg["smtp-port"])
>>
>> Note that this does actually send the email - 2 minutes later.
>
> Thanks for time and your effort Robert!
> In general the agent shouldn't got stuck if the broker is not able to
> send a notification email within a certain amount of time.
> I'm open a bug to track this. Adding Martin here.

https://bugzilla.redhat.com/1359059

>> So I tried:
>>
>>   $ telnet localhost 25
>>   Trying ::1...
>>
>> which hung, and a little bell went off in my brain...
>>
>> After changing /etc/hosts from:
>>
>> 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
>> ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
>>
>> to
>>
>> 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
>> ::1         localhost6 localhost6.localdomain6
>>
>> localhost resolves to 127.0.0.1, the delay is gone, and everything is fine.
>
> We are seeing similar reports regarding ip4/ip6 issues also migrating on 4.0
> See also http://lists.ovirt.org/pipermail/users/2016-June/040578.html and
> https://bugzilla.redhat.com/show_bug.cgi?id=1358530
>
> Adding Oved here.
>
>> I don't want to update /etc/hosts on each host. Is there somewhere I can
>> edit the broker config for mail?
>
> The shortest option is to edit broker.conf inside the configuration
> volume on the hosted-engine storage domain but it's a bit tricky and
> also potentially dangerous if not well done.
> We have an RFE about letting you reconfigure it from the engine, for
> now, if you are brave enough, please try something like this.
>
> dir=`mktemp -d` && cd $dir
> mnt_point=/rhev/data-center/mnt/192.168.1.115:_Virtual_ext35u36 #
> pleace with your local mount point
> systemctl stop ovirt-ha-broker # on all the hosts!
> sdUUID_line=$(grep sdUUID /etc/ovirt-hosted-engine/hosted-engine.conf)
> sdUUID=${sdUUID_line:7:36}
> conf_volume_UUID_line=$(grep conf_volume_UUID
> /etc/ovirt-hosted-engine/hosted-engine.conf)
> conf_volume_UUID=${conf_volume_UUID_line:17:36}
> conf_image_UUID_line=$(grep conf_image_UUID
> /etc/ovirt-hosted-engine/hosted-engine.conf)
> conf_image_UUID=${conf_image_UUID_line:16:36}
> sudo -u vdsm dd
> if=$mnt_point/$sdUUID/images/$conf_image_UUID/$conf_volume_UUID
> 2>/dev/null| tar -xvf -
> # here you have to edit the locally extracted broker.conf
> tar -cO * | sudo -u vdsm dd
> of=$mnt_point/$sdUUID/images/$conf_image_UUID/$conf_volume_UUID
> systemctl restart ovirt-ha-agent # on all the hosts
>
> I strongly advice to take a backup before editing.
>
>> Robert
>>
>> --
>> Senior Software Engineer @ Parsons