Hi, I have started a new installation as specified in the 3.4.1 release notes (fresh Fedora 19 install, yum localinstall http://resources.ovirt.org/pub/yum-repo/ovirt-release34.rpm). This is failing in the step "Configuring the management bridge". Based on the vdsm.log, it appears I am hitting: *Bug 988995* <https://bugzilla.redhat.com/show_bug.cgi?id=988995> -vdsm multipath.py restarts mutipathd, cutting the branch vdsm sits on "multipath -F" is returning "invalid keyword: getuid_callout" and it appears that this is causing vdsm-tool to abort (although the command exit status is 0 and the bug report says that those are only harmless warnings). There is no workaround stated in that bug report. Help? -Bob

I could really use some help on this one. My efforts to debug VDSM via instrumenting the python code are not working - the compiled code must be cached somehow. Something is wrong with the way the multipathd service is being restarted. It doesn't look too me that systemctl is even being called for it. Thanks, Bob On May 13, 2014 1:12 PM, "Bob Doolittle" <bob(a)doolittle.us.com&gt; wrote: > > Maybe this isn't the actual problem after all. > > I replaced /sbin/multipath with a script runs the old version, but the > suppresses those errors and returns exit status 0. > But "vdsm-tool service-reload multipathd" is still failing and I don't > know why. > > I have attached my vdsm.log file. > > Any guidance appreciated. I'll try digging through the python code for > service.py and see if I can catch it when the multipath configuration is in > place to see the exact issue. > > -Bob > > On 05/13/2014 12:27 PM, Bob Doolittle wrote: > > Hi, > > I have started a new installation as specified in the 3.4.1 release notes > (fresh Fedora 19 install, yum localinstall > http://resources.ovirt.org/pub/yum-repo/ovirt-release34.rpm). > > This is failing in the step "Configuring the management bridge". > > Based on the vdsm.log, it appears I am hitting: > Bug 988995 - vdsm multipath.py restarts mutipathd, cutting the branch vdsm > sits on > > "multipath -F" is returning "invalid keyword: getuid_callout" and it > appears that this is causing vdsm-tool to abort (although the command exit > status is 0 and the bug report says that those are only harmless warnings). > > There is no workaround stated in that bug report. > > Help? > > -Bob > > _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Thanks John. When hosted-engine aborts, it uninstalls everything. So there is no webadmin available. I've tried modifying the VDSM python code (e.g. /usr/share/vdsm/storage/multipath.py and /usr/lib64/python2.7/site-packages/vdsm/tool/service.py) to see/work around what's going wrong, but oddly VDSM does not seem to be affected. I suspect the original code (or bytecodes) are cached somewhere. Restarting vdsmd service has no effect. I'd really appreciate some insight there so I can work around it. Does oVirt 3.4.1 work more smoothly with F20? I chose F19 because it's more stable at this point, thinking that things would be more likely to work smoothly. It's not turning out that way... I'm willing to start over with F20 if there's no path forward.

-Bob On 05/14/2014 07:25 PM, John Taylor wrote: > Bob, > I remember something like this with an all-in-one install a while back > where that error showed up, but it was kind of red herring with > multipath because the real problem was ovirtmgmt bridge didn't get > created. And that was a known problem I think with f19 see > http://www.ovirt.org/OVirt_3.4_TestDay > "Important Note: Known Fedora 19 bug: If the ovirtmgmt bridge is not > successfully installed during initial host-setup, manually click on > the host, setup networks, and add the ovirtmgmt bridge. " > > If you can get to webadmin could you try to setup ovirtmgmt manually > on the host. > > -John > > On Wed, May 14, 2014 at 6:58 AM, Bob Doolittle <bob(a)doolittle.us.com&gt; wrote: >> I could really use some help on this one. My efforts to debug VDSM via >> instrumenting the python code are not working - the compiled code must be >> cached somehow. >> >> Something is wrong with the way the multipathd service is being restarted. >> It doesn't look too me that systemctl is even being called for it. >> >> Thanks, >> Bob >> >> On May 13, 2014 1:12 PM, "Bob Doolittle" <bob(a)doolittle.us.com&gt; wrote: >>> Maybe this isn't the actual problem after all. >>> >>> I replaced /sbin/multipath with a script runs the old version, but the >>> suppresses those errors and returns exit status 0. >>> But "vdsm-tool service-reload multipathd" is still failing and I don't >>> know why. >>> >>> I have attached my vdsm.log file. >>> >>> Any guidance appreciated. I'll try digging through the python code for >>> service.py and see if I can catch it when the multipath configuration is in >>> place to see the exact issue. >>> >>> -Bob >>> >>> On 05/13/2014 12:27 PM, Bob Doolittle wrote: >>> >>> Hi, >>> >>> I have started a new installation as specified in the 3.4.1 release notes >>> (fresh Fedora 19 install, yum localinstall >>> http://resources.ovirt.org/pub/yum-repo/ovirt-release34.rpm). >>> >>> This is failing in the step "Configuring the management bridge". >>> >>> Based on the vdsm.log, it appears I am hitting: >>> Bug 988995 - vdsm multipath.py restarts mutipathd, cutting the branch vdsm >>> sits on >>> >>> "multipath -F" is returning "invalid keyword: getuid_callout" and it >>> appears that this is causing vdsm-tool to abort (although the command exit >>> status is 0 and the bug report says that those are only harmless warnings). >>> >>> There is no workaround stated in that bug report. >>> >>> Help? >>> >>> -Bob >>> >>> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On Wed, May 14, 2014 at 08:06:00PM -0400, Bob Doolittle wrote: > Thanks John. > > When hosted-engine aborts, it uninstalls everything. > So there is no webadmin available. > > I've tried modifying the VDSM python code (e.g. > /usr/share/vdsm/storage/multipath.py and > /usr/lib64/python2.7/site-packages/vdsm/tool/service.py) to see/work around > what's going wrong, but oddly VDSM does not seem to be affected. I suspect > the original code (or bytecodes) are cached somewhere. I do not think that this is the issue, but you can remove all trace of *.pyc/*.pyo to make sure this is not the case.

> Restarting vdsmd > service has no effect. I'd really appreciate some insight there so I can > work around it. What have you changed exactly? Where? If you add a plain syntax error to the script, does it still run?

> Does oVirt 3.4.1 work more smoothly with F20? I chose F19 because it's more > stable at this point, thinking that things would be more likely to work > smoothly. It's not turning out that way... I'm willing to start over with > F20 if there's no path forward. Vdsm works fine on 3.4.1, but I do not know about hosted-engine. Engine itself is known not to work on F20.

Il 15/05/2014 15:55, Bob Doolittle ha scritto: > If I run "vdsm-tool service-reload multipathd" by hand then I see the log I'd expect: > reload multipathd.service If this command works, hosted-engine setup should not fail on it. The setup doesn't touch anything related to multipathd.

Il 15/05/2014 16:09, Bob Doolittle ha scritto: > > On 05/15/2014 09:59 AM, Sandro Bonazzola wrote: >> Il 15/05/2014 15:55, Bob Doolittle ha scritto: >>> If I run "vdsm-tool service-reload multipathd" by hand then I see the log I'd expect: >>> reload multipathd.service >> If this command works, hosted-engine setup should not fail on it. >> The setup doesn't touch anything related to multipathd. > > Sorry but that is not strictly correct. > You can see the code in multipath.py (setupMultipath()) that writes a new multipath.conf file just before it tries to restart the service. > It looks like it's trying to modify the scsi_id_path. > > So when I run it by hand, I am using the original multipath.conf file. When it is run during hosted-engine setup, it is with a modified version. > My efforts in modifying the python have been to try to capture that modified version, so that I can exercise it by hand and see what's wrong. If something is configuring it, it must be vdsm-tool:

In hosted-engine setup code there's a call to "vdsm-tool configure --force".

Maybe this isn't the actual problem after all. I replaced /sbin/multipath with a script runs the old version, but the suppresses those errors and returns exit status 0. But "vdsm-tool service-reload multipathd" is still failing and I don't know why.

I have attached my vdsm.log file. Any guidance appreciated. I'll try digging through the python code for service.py and see if I can catch it when the multipath configuration is in place to see the exact issue.