5:06 a.m.
Hi,
On yesterday's vdsm weekly call, we were discussing the need of making
Python 3 vdsm RPM packages.
Some facts:
- it doesn't make a lot sense to spend much time on trying to package
everything - it's completely impossible i.e. to run vdsm without having
'sanlock' module
- our current vdsm.spec file is crap
Two non-exclusive propositions were raised:
- let's try to make a quick-and-dirty patch, that will completely
overwrite the existing 'vdsm.spec' (effectively making it Python 3-only)
for testing purposes, and maintain it for a while
- in the meantime, let's write a completely new, clean and beautiful
spec file in an package-by-package, incremental manner, (also Python
3-only) that would eventually substitute the original one
The quick-and-dirty spec file would be completely unsupported by CI. The
new one would get a proper CI sub-stage in 'build-artifacts' stage.
The steps needed to be done are:
- prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds
- prepare the new spec file (for now including only 'vdsm-common' package)
- split 'build-artifacts' stage into 'build-py27' and 'build-py36'
sub-stages (the latter currently running on fc28 only)
The only package we can start with, when making the new spec file, is
'vdsm-common', as it doesn't depend on anything else (or at least I hope
so...).
There were also propositions about how to change the new spec file in
regard to the old one (like making 'vdsm' package a meta-package). This
is a good time for these propositions to be raised, reviewed and
documented (something like this maybe?
https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72...),
so we can align the new spec file as we build it.
I can lay the groundwork by doing the autotools/Makefiles and
'build-artifacts' splitting. Gal Zaidman agreed on starting to work on
the new spec file. Milan mentioned, that he had something like the
quick-and-dirty patch, maybe he can share it with us.
Questions, comments are welcome.
Regards, Marcin
5:17 a.m.
On Thu, Feb 28, 2019 at 1:06 PM Marcin Sobczyk <msobczyk(a)redhat.com> wrote:
Hi,
On yesterday's vdsm weekly call, we were discussing the need of making
Python 3 vdsm RPM packages.
Some facts:
- it doesn't make a lot sense to spend much time on trying to package
everything - it's completely impossible i.e. to run vdsm without having
'sanlock' module
localfs storage works without sanlock, so we have a way to test the entire
system without sanlock. We can make sanlock configurable to be able
to build and run vdsm without it with only local storage.
But I think completing the sanlock port to python 3 will be much less work.
- our current vdsm.spec file is crap
Two non-exclusive propositions were raised:
- let's try to make a quick-and-dirty patch, that will completely
overwrite the existing 'vdsm.spec' (effectively making it Python 3-only)
for testing purposes, and maintain it for a while
- in the meantime, let's write a completely new, clean and beautiful
spec file in an package-by-package, incremental manner, (also Python
3-only) that would eventually substitute the original one
The quick-and-dirty spec file would be completely unsupported by CI. The
new one would get a proper CI sub-stage in 'build-artifacts' stage.
The steps needed to be done are:
- prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds
- prepare the new spec file (for now including only 'vdsm-common' package)
- split 'build-artifacts' stage into 'build-py27' and
'build-py36'
sub-stages (the latter currently running on fc28 only)
The only package we can start with, when making the new spec file, is
'vdsm-common', as it doesn't depend on anything else (or at least I hope
so...).
There were also propositions about how to change the new spec file in
regard to the old one (like making 'vdsm' package a meta-package). This
is a good time for these propositions to be raised, reviewed and
documented (something like this maybe?
https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72...),
so we can align the new spec file as we build it.
I can lay the groundwork by doing the autotools/Makefiles and
'build-artifacts' splitting. Gal Zaidman agreed on starting to work on
the new spec file. Milan mentioned, that he had something like the
quick-and-dirty patch, maybe he can share it with us.
Questions, comments are welcome.
Regards, Marcin
Thanks for the summary!
Nir
5:30 a.m.
On Thu, Feb 28, 2019 at 1:07 PM Marcin Sobczyk <msobczyk(a)redhat.com> wrote:
Hi,
On yesterday's vdsm weekly call, we were discussing the need of making
Python 3 vdsm RPM packages.
Some facts:
- it doesn't make a lot sense to spend much time on trying to package
everything - it's completely impossible i.e. to run vdsm without having
'sanlock' module
- our current vdsm.spec file is crap
Two non-exclusive propositions were raised:
- let's try to make a quick-and-dirty patch, that will completely
overwrite the existing 'vdsm.spec' (effectively making it Python 3-only)
for testing purposes, and maintain it for a while
- in the meantime, let's write a completely new, clean and beautiful
spec file in an package-by-package, incremental manner, (also Python
3-only) that would eventually substitute the original one
I'm not sure I understand that second option.
I am afraid of fresh starts; I'd very much prefer to start from the
sh*tty thing we have, and evolve it. A lot of time, re-writing a piece
of software is tempting, but existing code is imbued with knowledge of
past problems, which is often forgotten when you do a hard cut.
Cleaning %files should be an easy first step; I think that Gal's
jinja-based generation of py2/py3 packages is sane.
Can you explain why not just to carry these patches over?
>
> The quick-and-dirty spec file would be completely unsupported by CI. The
> new one would get a proper CI sub-stage in 'build-artifacts' stage.
>
> The steps needed to be done are:
>
> - prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds
> - prepare the new spec file (for now including only 'vdsm-common' package)
> - split 'build-artifacts' stage into 'build-py27' and
'build-py36'
> sub-stages (the latter currently running on fc28 only)
>
> The only package we can start with, when making the new spec file, is
> 'vdsm-common', as it doesn't depend on anything else (or at least I hope
> so...).
>
> There were also propositions about how to change the new spec file in
> regard to the old one (like making 'vdsm' package a meta-package). This
> is a good time for these propositions to be raised, reviewed and
> documented (something like this maybe?
>
https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72...),
> so we can align the new spec file as we build it.
>
> I can lay the groundwork by doing the autotools/Makefiles and
> 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on
> the new spec file. Milan mentioned, that he had something like the
> quick-and-dirty patch, maybe he can share it with us.
>
> Questions, comments are welcome.
>
> Regards, Marcin
>
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org
> To unsubscribe send an email to devel-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MFZHLJA46QM...
5:42 a.m.
On 2/28/19 12:30 PM, Dan Kenigsberg wrote:
On Thu, Feb 28, 2019 at 1:07 PM Marcin Sobczyk
<msobczyk(a)redhat.com> wrote:
> Hi,
>
> On yesterday's vdsm weekly call, we were discussing the need of making
> Python 3 vdsm RPM packages.
>
> Some facts:
>
> - it doesn't make a lot sense to spend much time on trying to package
> everything - it's completely impossible i.e. to run vdsm without having
> 'sanlock' module
> - our current vdsm.spec file is crap
>
> Two non-exclusive propositions were raised:
>
> - let's try to make a quick-and-dirty patch, that will completely
> overwrite the existing 'vdsm.spec' (effectively making it Python 3-only)
> for testing purposes, and maintain it for a while
> - in the meantime, let's write a completely new, clean and beautiful
> spec file in an package-by-package, incremental manner, (also Python
> 3-only) that would eventually substitute the original one
I'm not sure I understand that second option.
I am afraid of fresh starts; I'd very much prefer to start from the
sh*tty thing we have, and evolve it. A lot of time, re-writing a piece
of software is tempting, but existing code is imbued with knowledge of
past problems, which is often forgotten when you do a hard cut.
Cleaning %files should be an easy first step; I think that Gal's
jinja-based generation of py2/py3 packages is sane.
Can you explain why not just to carry these patches over?
AFAIK, 4.4 will be
RHEL8-only. This means, we will drop Python 2 completely.
Trying to make the existing spec file work with both Python 2 and Python
3 is a wasted effort.
It also complicates things a lot (new build dependency, 2 layers of
preprocessing) unnecessarily.
I'm pretty confident, that introducing the new spec file package by
package, will work for us - it's not Python code, errors will probably
emerge quickly.
+, we would still have the original spec, until we're done, to have a
reference point.
>
>> The quick-and-dirty spec file would be completely unsupported by CI. The
>> new one would get a proper CI sub-stage in 'build-artifacts' stage.
>>
>> The steps needed to be done are:
>>
>> - prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds
>> - prepare the new spec file (for now including only 'vdsm-common'
package)
>> - split 'build-artifacts' stage into 'build-py27' and
'build-py36'
>> sub-stages (the latter currently running on fc28 only)
>>
>> The only package we can start with, when making the new spec file, is
>> 'vdsm-common', as it doesn't depend on anything else (or at least I
hope
>> so...).
>>
>> There were also propositions about how to change the new spec file in
>> regard to the old one (like making 'vdsm' package a meta-package). This
>> is a good time for these propositions to be raised, reviewed and
>> documented (something like this maybe?
>>
https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72...),
>> so we can align the new spec file as we build it.
>>
>> I can lay the groundwork by doing the autotools/Makefiles and
>> 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on
>> the new spec file. Milan mentioned, that he had something like the
>> quick-and-dirty patch, maybe he can share it with us.
>>
>> Questions, comments are welcome.
>>
>> Regards, Marcin
>>
>> _______________________________________________
>> Devel mailing list -- devel(a)ovirt.org
>> To unsubscribe send an email to devel-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MFZHLJA46QM...
5:54 a.m.
On Thu, Feb 28, 2019 at 1:30 PM Dan Kenigsberg <danken(a)redhat.com> wrote:
On Thu, Feb 28, 2019 at 1:07 PM Marcin Sobczyk
<msobczyk(a)redhat.com>
wrote:
>
> Hi,
>
> On yesterday's vdsm weekly call, we were discussing the need of making
> Python 3 vdsm RPM packages.
>
> Some facts:
>
> - it doesn't make a lot sense to spend much time on trying to package
> everything - it's completely impossible i.e. to run vdsm without having
> 'sanlock' module
> - our current vdsm.spec file is crap
>
> Two non-exclusive propositions were raised:
>
> - let's try to make a quick-and-dirty patch, that will completely
> overwrite the existing 'vdsm.spec' (effectively making it Python 3-only)
> for testing purposes, and maintain it for a while
> - in the meantime, let's write a completely new, clean and beautiful
> spec file in an package-by-package, incremental manner, (also Python
> 3-only) that would eventually substitute the original one
I'm not sure I understand that second option.
I am afraid of fresh starts; I'd very much prefer to start from the
sh*tty thing we have, and evolve it. A lot of time, re-writing a piece
of software is tempting, but existing code is imbued with knowledge of
past problems, which is often forgotten when you do a hard cut.
True, but if you pick a small enough component, replacing bad component
with good one is much quicker.
Cleaning %files should be an easy first step;
This is a good idea anyway.
I think that Gal's
jinja-based generation of py2/py3 packages is sane.
Can you explain why not just to carry these patches over?
Adding jinna to the mix will only make the spec harder to work with, we
will have
spec + shell + jinna + autoconf in the same file. And when we delete the
python 2
support we will have to live with that complexity that serve nothing.
We need now:
- python 2 for rhel 7
- python 2 for fedora
- python 3 for rhel 8
- python 3 for fedora
Once we have working python 3 versions, we can drop the python 2 builds,
so we will have a single spec with:
- pyhton 3 for rhel 8
- pyhton 3 for fedora
So investing time and energy on having "smart" spec that support both
python 2
and 3 is waste of effort.
Idealy the new spec will be static, no code generation, no code. All code
(e.g. install/upgrade scripts) should be in separate files that can be
tested easily.
> The quick-and-dirty spec file would be completely unsupported by CI. The
> new one would get a proper CI sub-stage in 'build-artifacts' stage.
>
> The steps needed to be done are:
>
> - prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM
builds
> - prepare the new spec file (for now including only 'vdsm-common'
package)
> - split 'build-artifacts' stage into 'build-py27' and
'build-py36'
> sub-stages (the latter currently running on fc28 only)
>
> The only package we can start with, when making the new spec file, is
> 'vdsm-common', as it doesn't depend on anything else (or at least I
hope
> so...).
>
> There were also propositions about how to change the new spec file in
> regard to the old one (like making 'vdsm' package a meta-package). This
> is a good time for these propositions to be raised, reviewed and
> documented (something like this maybe?
>
https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72...
),
> so we can align the new spec file as we build it.
>
> I can lay the groundwork by doing the autotools/Makefiles and
> 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on
> the new spec file. Milan mentioned, that he had something like the
> quick-and-dirty patch, maybe he can share it with us.
>
> Questions, comments are welcome.
>
> Regards, Marcin
>
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org
> To unsubscribe send an email to devel-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MFZHLJA46QM...
6 a.m.
On 2/28/19 12:30 PM, Dan Kenigsberg wrote:
I am afraid of fresh starts; I'd very much prefer to start from
the
sh*tty thing we have, and evolve it. A lot of time, re-writing a piece
of software is tempting, but existing code is imbued with knowledge of
past problems, which is often forgotten when you do a hard cut.
In general, this is very true.
However, in the *specific case* of the spec file, it is mostly an
aggregation of fixes, hacks and workarounds
arised from contingent issues. Or at very least this is my experience
with the spec file.
My point is the spec file is mostly a snapshot of the fixes needed for a
given set of supported OSes.[1]
There isn't much to salvage here, and for what's worth keeping, git
history is the best record, and this is not going to be lost.
For these reasons, I support Marcin's plan to start afresh side by side
with a new spec file, using the old one as reference.
I trust Marcin (and us) to be careful and look back often at the old
spec file when writing the new one, to avoid forgetting
the lessons of the past.
+++
[1] the current spec file looks much more like a scratchpad full of
scribbled notes than a careful curated chronicle imbued with knowledge.
Bests,
--
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh
7:02 a.m.
Marcin Sobczyk <msobczyk(a)redhat.com> writes:
Hi,
On yesterday's vdsm weekly call, we were discussing the need of making
Python 3 vdsm RPM packages.
Some facts:
- it doesn't make a lot sense to spend much time on trying to package
everything - it's completely impossible i.e. to run vdsm without
having 'sanlock' module
- our current vdsm.spec file is crap
Two non-exclusive propositions were raised:
- let's try to make a quick-and-dirty patch, that will completely
overwrite the existing 'vdsm.spec' (effectively making it Python
3-only) for testing purposes, and maintain it for a while
- in the meantime, let's write a completely new, clean and beautiful
spec file in an package-by-package, incremental manner, (also Python
3-only) that would eventually substitute the original one
The quick-and-dirty spec file would be completely unsupported by
CI. The new one would get a proper CI sub-stage in 'build-artifacts'
stage.
The steps needed to be done are:
- prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds
- prepare the new spec file (for now including only 'vdsm-common' package)
- split 'build-artifacts' stage into 'build-py27' and
'build-py36'
sub-stages (the latter currently running on fc28 only)
And solve #/usr/bin/python* issue.
The only package we can start with, when making the new spec file,
is
'vdsm-common', as it doesn't depend on anything else (or at least I
hope so...).
There were also propositions about how to change the new spec file in
regard to the old one (like making 'vdsm' package a
meta-package). This is a good time for these propositions to be
raised, reviewed and documented (something like this maybe?
https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72...),
so we can align the new spec file as we build it.
I can lay the groundwork by doing the autotools/Makefiles and
'build-artifacts' splitting. Gal Zaidman agreed on starting to work on
the new spec file. Milan mentioned, that he had something like the
quick-and-dirty patch, maybe he can share it with us.
I can try to post a patch with a q&d Python 3 vdsm.spec.in file; maybe
later today.
> Questions, comments are welcome.
>
> Regards, Marcin
2:39 p.m.
Hi,
I've posted a series of patches:
https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+statu...
They do a couple of things:
* add an '--enable-python3-build' argument to 'autogen.sh'
* modify 'Makefile.am' so it uses either 'vdsm.spec.in' or
'vdsm-py3.spec.in' file to build RPMs
* preserve the 'vdsm.spec[.in]' filename in dist archives for both py2
and py3 packaging
* fix shebang lines in a dist-hook, so that py3 packages will contain
only '#!/usr/bin/python3' shebangs
* prepare the CI for building py3 packages on fc28
So actually, if we replace the dummy 'vdsm-py3.spec.in' file in
https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the
nasty, hacky one, or the shiny, new one), we're ready to build py3 packages.
Reviews are more than welcome.
Regards, Marcin
On 2/28/19 2:02 PM, Milan Zamazal wrote:
Marcin Sobczyk <msobczyk(a)redhat.com> writes:
> Hi,
>
> On yesterday's vdsm weekly call, we were discussing the need of making
> Python 3 vdsm RPM packages.
>
> Some facts:
>
> - it doesn't make a lot sense to spend much time on trying to package
> everything - it's completely impossible i.e. to run vdsm without
> having 'sanlock' module
> - our current vdsm.spec file is crap
>
> Two non-exclusive propositions were raised:
>
> - let's try to make a quick-and-dirty patch, that will completely
> overwrite the existing 'vdsm.spec' (effectively making it Python
> 3-only) for testing purposes, and maintain it for a while
> - in the meantime, let's write a completely new, clean and beautiful
> spec file in an package-by-package, incremental manner, (also Python
> 3-only) that would eventually substitute the original one
>
> The quick-and-dirty spec file would be completely unsupported by
> CI. The new one would get a proper CI sub-stage in 'build-artifacts'
> stage.
>
> The steps needed to be done are:
>
> - prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds
> - prepare the new spec file (for now including only 'vdsm-common' package)
> - split 'build-artifacts' stage into 'build-py27' and
'build-py36'
> sub-stages (the latter currently running on fc28 only)
And solve #/usr/bin/python* issue.
> The only package we can start with, when making the new spec file, is
> 'vdsm-common', as it doesn't depend on anything else (or at least I
> hope so...).
>
> There were also propositions about how to change the new spec file in
> regard to the old one (like making 'vdsm' package a
> meta-package). This is a good time for these propositions to be
> raised, reviewed and documented (something like this maybe?
>
https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72...),
> so we can align the new spec file as we build it.
>
> I can lay the groundwork by doing the autotools/Makefiles and
> 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on
> the new spec file. Milan mentioned, that he had something like the
> quick-and-dirty patch, maybe he can share it with us.
I can try to post a patch with a q&d Python 3 vdsm.spec.in file; maybe
later today.
> Questions, comments are welcome.
>
> Regards, Marcin
7:39 a.m.
Marcin Sobczyk <msobczyk(a)redhat.com> writes:
I've posted a series of patches:
https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+statu...
They do a couple of things:
* add an '--enable-python3-build' argument to 'autogen.sh'
* modify 'Makefile.am' so it uses either 'vdsm.spec.in' or
'vdsm-py3.spec.in' file to build RPMs
* preserve the 'vdsm.spec[.in]' filename in dist archives for both py2
and py3 packaging
* fix shebang lines in a dist-hook, so that py3 packages will contain
only '#!/usr/bin/python3' shebangs
* prepare the CI for building py3 packages on fc28
Hi Marcin, cool, thank you!
So actually, if we replace the dummy 'vdsm-py3.spec.in' file
in
https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the
nasty, hacky one, or the shiny, new one), we're ready to build py3
packages.
I've made a dirty one: https://gerrit.ovirt.org/98150 (you can simply
cherry pick the to wherever you need).
However, it doesn't work, it's still built with Python 2 on my Fedora
28. I can see a lot of warning such as
DEPRECATION WARNING: python2 invoked with /usr/bin/python.
Use /usr/bin/python3 or /usr/bin/python2
/usr/bin/python will be removed or switched to Python 3 in the future.
I can see that configure.ac in your patches still uses Python 2. Here
is what I used in my private hacks to change that. Something similar
should be done, depending on --enable-python3-build presence, and with
properly detected Python versions.
Would you like to make the necessary changes or should I make them
myself?
modified configure.ac
@@ -48,7 +48,7 @@ AM_INIT_AUTOMAKE([-Wno-portability])
# Checking for build tools
AC_PROG_LN_S
-AM_PATH_PYTHON([2.6])
+AM_PATH_PYTHON([3.6])
AC_ARG_ENABLE(
[hooks],
@@ -349,13 +349,13 @@ if test "$PYTHON3_SUPPORT" = "1"; then
fi
# Checking for python-devel
-AC_PATH_PROG([PYTHON_CONFIG], [python-config])
+AC_PATH_PROG([PYTHON_CONFIG], [python3-config])
if test "x$PYTHON_CONFIG" = "x"; then
- AC_MSG_ERROR([python-devel not found, please install it.])
+ AC_MSG_ERROR([python3-devel not found, please install it.])
fi
# Checking for python modules (sorted, please keep in order)
-AX_PYTHON_MODULE([six], [fatal], python2)
+AX_PYTHON_MODULE([six], [fatal], python3)
# External programs (sorted, please keep in order)
AC_PATH_PROG([CAT_PATH], [cat], [/bin/cat])
8:46 a.m.
On 3/1/19 2:39 PM, Milan Zamazal wrote:
Marcin Sobczyk <msobczyk(a)redhat.com> writes:
> I've posted a series of patches:
> https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+statu...
>
> They do a couple of things:
>
> * add an '--enable-python3-build' argument to 'autogen.sh'
> * modify 'Makefile.am' so it uses either 'vdsm.spec.in' or
> 'vdsm-py3.spec.in' file to build RPMs
> * preserve the 'vdsm.spec[.in]' filename in dist archives for both py2
> and py3 packaging
> * fix shebang lines in a dist-hook, so that py3 packages will contain
> only '#!/usr/bin/python3' shebangs
> * prepare the CI for building py3 packages on fc28
Hi Marcin, cool, thank you!
> So actually, if we replace the dummy 'vdsm-py3.spec.in' file in
> https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the
> nasty, hacky one, or the shiny, new one), we're ready to build py3
> packages.
I've made a dirty one: https://gerrit.ovirt.org/98150 (you can simply
cherry pick the to wherever you need).
However, it doesn't work, it's still built with Python 2 on my Fedora
28. I can see a lot of warning such as
DEPRECATION WARNING: python2 invoked with /usr/bin/python.
Use /usr/bin/python3 or /usr/bin/python2
/usr/bin/python will be removed or switched to Python 3 in the future.
I can see that configure.ac in your patches still uses Python 2. Here
is what I used in my private hacks to change that. Something similar
should be done, depending on --enable-python3-build presence, and with
properly detected Python versions.
Good point - I'm currently working on resolving that. It got kinda
complicated but will keep you posted as soon as I get some results.
>
> Would you like to make the necessary changes or should I make them
> myself?
>
> modified configure.ac
> @@ -48,7 +48,7 @@ AM_INIT_AUTOMAKE([-Wno-portability])
>
> # Checking for build tools
> AC_PROG_LN_S
> -AM_PATH_PYTHON([2.6])
> +AM_PATH_PYTHON([3.6])
>
> AC_ARG_ENABLE(
> [hooks],
> @@ -349,13 +349,13 @@ if test "$PYTHON3_SUPPORT" = "1"; then
> fi
>
> # Checking for python-devel
> -AC_PATH_PROG([PYTHON_CONFIG], [python-config])
> +AC_PATH_PROG([PYTHON_CONFIG], [python3-config])
> if test "x$PYTHON_CONFIG" = "x"; then
> - AC_MSG_ERROR([python-devel not found, please install it.])
> + AC_MSG_ERROR([python3-devel not found, please install it.])
> fi
>
> # Checking for python modules (sorted, please keep in order)
> -AX_PYTHON_MODULE([six], [fatal], python2)
> +AX_PYTHON_MODULE([six], [fatal], python3)
>
> # External programs (sorted, please keep in order)
> AC_PATH_PROG([CAT_PATH], [cat], [/bin/cat])
6:19 a.m.
Hi,
it took a while... after trying to grasp "the big picture", I came up
with these:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
The old topic:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
was rebased (not yet 100%) on top of 'build-system-overhaul'.
I've tried to implement a solution, that would clean up our
configure.ac/Makefiles, make Python 3 packaging possible, make running
tests with multiple interpreter versions possible for the developers, etc.
I know, that I will get "crazy looks" for writing m4 macros, but I
really believe having stuff like 'VDSM_PY_SUPPORTED_MAJOR_VERSION' et
al. will make developers' lives easier.
I'm currently trying to address issues and comments raised by Milan, I
would definitely like to get others opinions on these.
Regards, Marcin
On 3/4/19 3:46 PM, Marcin Sobczyk wrote:
> On 3/1/19 2:39 PM, Milan Zamazal wrote:
>> Marcin Sobczyk <msobczyk(a)redhat.com> writes:
>>
>>> I've posted a series of patches:
>>>
https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+statu...
>>>
>>>
>>> They do a couple of things:
>>>
>>> * add an '--enable-python3-build' argument to 'autogen.sh'
>>> * modify 'Makefile.am' so it uses either 'vdsm.spec.in' or
>>> 'vdsm-py3.spec.in' file to build RPMs
>>> * preserve the 'vdsm.spec[.in]' filename in dist archives for both
>>> py2
>>> and py3 packaging
>>> * fix shebang lines in a dist-hook, so that py3 packages will contain
>>> only '#!/usr/bin/python3' shebangs
>>> * prepare the CI for building py3 packages on fc28
>> Hi Marcin, cool, thank you!
>>
>>> So actually, if we replace the dummy 'vdsm-py3.spec.in' file in
>>> https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the
>>> nasty, hacky one, or the shiny, new one), we're ready to build py3
>>> packages.
>> I've made a dirty one: https://gerrit.ovirt.org/98150 (you can simply
>> cherry pick the to wherever you need).
>>
>> However, it doesn't work, it's still built with Python 2 on my Fedora
>> 28. I can see a lot of warning such as
>>
>> DEPRECATION WARNING: python2 invoked with /usr/bin/python.
>> Use /usr/bin/python3 or /usr/bin/python2
>> /usr/bin/python will be removed or switched to Python 3 in the
>> future.
>>
>> I can see that configure.ac in your patches still uses Python 2. Here
>> is what I used in my private hacks to change that. Something similar
>> should be done, depending on --enable-python3-build presence, and with
>> properly detected Python versions.
>
> Good point - I'm currently working on resolving that. It got kinda
> complicated but will keep you posted as soon as I get some results.
>
>>
>> Would you like to make the necessary changes or should I make them
>> myself?
>>
>> modified configure.ac
>> @@ -48,7 +48,7 @@ AM_INIT_AUTOMAKE([-Wno-portability])
>> # Checking for build tools
>> AC_PROG_LN_S
>> -AM_PATH_PYTHON([2.6])
>> +AM_PATH_PYTHON([3.6])
>> AC_ARG_ENABLE(
>> [hooks],
>> @@ -349,13 +349,13 @@ if test "$PYTHON3_SUPPORT" = "1"; then
>> fi
>> # Checking for python-devel
>> -AC_PATH_PROG([PYTHON_CONFIG], [python-config])
>> +AC_PATH_PROG([PYTHON_CONFIG], [python3-config])
>> if test "x$PYTHON_CONFIG" = "x"; then
>> - AC_MSG_ERROR([python-devel not found, please install it.])
>> + AC_MSG_ERROR([python3-devel not found, please install it.])
>> fi
>> # Checking for python modules (sorted, please keep in order)
>> -AX_PYTHON_MODULE([six], [fatal], python2)
>> +AX_PYTHON_MODULE([six], [fatal], python3)
>> # External programs (sorted, please keep in order)
>> AC_PATH_PROG([CAT_PATH], [cat], [/bin/cat])
5:48 a.m.
Hi,
I've made a google document, that describes the changes that come with
the patches and the rationale behind them:
https://docs.google.com/document/d/1XBf9eP0lh3IeRzkQ3X-c191rMpOwsTTkX-eao...
Please note, that the patches do not reflect 100% what's described in
the doc - while writing the doc I came to conclusions, that some things
should be improved.
Since Milan told, that Python 3 packaging is very high priority for him
(even if the packages don't actually work yet), I would like to ask you
to get involved in reviewing the document.
After we establish a common ground, I will align my patches to reflect
what's in the doc.
P.S. Previous Markdown version was a pain to use so I just rewrote the
thing as a simple doc.
Thanks, Marcin
On 3/18/19 12:19 PM, Marcin Sobczyk wrote:
> Hi,
>
> it took a while... after trying to grasp "the big picture", I came up
> with these:
>
>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
>
>
> The old topic:
>
>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
>
>
> was rebased (not yet 100%) on top of 'build-system-overhaul'.
>
> I've tried to implement a solution, that would clean up our
> configure.ac/Makefiles, make Python 3 packaging possible, make running
> tests with multiple interpreter versions possible for the developers,
> etc.
> I know, that I will get "crazy looks" for writing m4 macros, but I
> really believe having stuff like 'VDSM_PY_SUPPORTED_MAJOR_VERSION' et
> al. will make developers' lives easier.
> I'm currently trying to address issues and comments raised by Milan, I
> would definitely like to get others opinions on these.
>
> Regards, Marcin
>
> On 3/4/19 3:46 PM, Marcin Sobczyk wrote:
>> On 3/1/19 2:39 PM, Milan Zamazal wrote:
>>> Marcin Sobczyk <msobczyk(a)redhat.com> writes:
>>>
>>>> I've posted a series of patches:
>>>>
https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+statu...
>>>>
>>>>
>>>> They do a couple of things:
>>>>
>>>> * add an '--enable-python3-build' argument to
'autogen.sh'
>>>> * modify 'Makefile.am' so it uses either 'vdsm.spec.in'
or
>>>> 'vdsm-py3.spec.in' file to build RPMs
>>>> * preserve the 'vdsm.spec[.in]' filename in dist archives for
>>>> both py2
>>>> and py3 packaging
>>>> * fix shebang lines in a dist-hook, so that py3 packages will
>>>> contain
>>>> only '#!/usr/bin/python3' shebangs
>>>> * prepare the CI for building py3 packages on fc28
>>> Hi Marcin, cool, thank you!
>>>
>>>> So actually, if we replace the dummy 'vdsm-py3.spec.in' file in
>>>> https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the
>>>> nasty, hacky one, or the shiny, new one), we're ready to build py3
>>>> packages.
>>> I've made a dirty one: https://gerrit.ovirt.org/98150 (you can simply
>>> cherry pick the to wherever you need).
>>>
>>> However, it doesn't work, it's still built with Python 2 on my
Fedora
>>> 28. I can see a lot of warning such as
>>>
>>> DEPRECATION WARNING: python2 invoked with /usr/bin/python.
>>> Use /usr/bin/python3 or /usr/bin/python2
>>> /usr/bin/python will be removed or switched to Python 3 in
>>> the future.
>>>
>>> I can see that configure.ac in your patches still uses Python 2. Here
>>> is what I used in my private hacks to change that. Something similar
>>> should be done, depending on --enable-python3-build presence, and with
>>> properly detected Python versions.
>>
>> Good point - I'm currently working on resolving that. It got kinda
>> complicated but will keep you posted as soon as I get some results.
>>
>>>
>>> Would you like to make the necessary changes or should I make them
>>> myself?
>>>
>>> modified configure.ac
>>> @@ -48,7 +48,7 @@ AM_INIT_AUTOMAKE([-Wno-portability])
>>> # Checking for build tools
>>> AC_PROG_LN_S
>>> -AM_PATH_PYTHON([2.6])
>>> +AM_PATH_PYTHON([3.6])
>>> AC_ARG_ENABLE(
>>> [hooks],
>>> @@ -349,13 +349,13 @@ if test "$PYTHON3_SUPPORT" = "1";
then
>>> fi
>>> # Checking for python-devel
>>> -AC_PATH_PROG([PYTHON_CONFIG], [python-config])
>>> +AC_PATH_PROG([PYTHON_CONFIG], [python3-config])
>>> if test "x$PYTHON_CONFIG" = "x"; then
>>> - AC_MSG_ERROR([python-devel not found, please install it.])
>>> + AC_MSG_ERROR([python3-devel not found, please install it.])
>>> fi
>>> # Checking for python modules (sorted, please keep in order)
>>> -AX_PYTHON_MODULE([six], [fatal], python2)
>>> +AX_PYTHON_MODULE([six], [fatal], python3)
>>> # External programs (sorted, please keep in order)
>>> AC_PATH_PROG([CAT_PATH], [cat], [/bin/cat])
4:23 a.m.
Hi,
a quick follow up on my py3-packaging patches.
I've split them into parts, because there's a lot of them, and some of
the older patches needed to be rebased on top of new stuff.
Two issues arised when working on the patches.
The first one is trying to support both py36 and py37 on Fedora 29.
We use 'automation/*packages*' files to decide what needs to be
installed in order to run the tests.
These files contain names of yum/dnf packages.
The thing is, that these packages are installed into
'/usr/lib/python3.7/site-packages'.
When trying to run py36 tests, they can't be found.
This can be fixed, by including the test dependencies in 'tox.ini' in
the 'deps' section.
This is how tox handles dependencies:
- if the package is already available in the OS, it uses that version
(py37 case)
- if the package is not available, it installs it to its local
environment with 'pip' (py36 case)
Listing the dependencies in the 'deps' section fixes the problem... with
one exception.
'selinux' python bindings are not available in PyPI - there's only the
yum/dnf package version.
A workaround would be to mark the tests that require 'selinux' with
something like 'test_on_target_python_only'.
I think we should focus on the packaging stuff right now, so I would
postpone the fix for this problem.
The first batch of patches [1] partially deals with the issue by a
workaround.
It limits the tested interpreter versions to "python2.7 python3" instead
of "python2.7 python3.6 python3.7".
The effect is, that if a developer runs Fedora 29, and has both py36 and
py37 installed, only py37 will be used by default (that's the one that
'python3' is pointing to).
CI is unaffected by this change (since there's only one py3 version anyway).
The second issue is byte-compiling our python modules.
Everything suggests that we're doing it twice (once by
'build-aux/py-compile' and the second time is done by
'brp-python-bytecompile' script [2]) as there are files like
'API.cpython-37.pyc' and 'API.cpython-37.opt-1.pyc' in the RPM build
root dir, and they are identical.
I'm still working on cleaning this mess up.
Other than that, there are patches that unify the makefile's test
targets [3] and the ones that add autogen.sh flags as described in the
design document [4], which are ready to be reviewed.
It would be nice if someone else played around a bit with the flags - I
use them all the time and they seem to work for me, but I'm already
occupied with the byte-compiling issue.
Thanks, Marcin
[1]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
[2]
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix...
[3]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
[4]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
On 3/28/19 11:48 AM, Marcin Sobczyk wrote:
> Hi,
>
> I've made a google document, that describes the changes that come with
> the patches and the rationale behind them:
>
>
https://docs.google.com/document/d/1XBf9eP0lh3IeRzkQ3X-c191rMpOwsTTkX-eao...
>
>
> Please note, that the patches do not reflect 100% what's described in
> the doc - while writing the doc I came to conclusions, that some
> things should be improved.
> Since Milan told, that Python 3 packaging is very high priority for
> him (even if the packages don't actually work yet), I would like to
> ask you to get involved in reviewing the document.
> After we establish a common ground, I will align my patches to reflect
> what's in the doc.
>
> P.S. Previous Markdown version was a pain to use so I just rewrote the
> thing as a simple doc.
>
> Thanks, Marcin
>
> On 3/18/19 12:19 PM, Marcin Sobczyk wrote:
>> Hi,
>>
>> it took a while... after trying to grasp "the big picture", I came up
>> with these:
>>
>>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
>>
>>
>> The old topic:
>>
>>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
>>
>>
>> was rebased (not yet 100%) on top of 'build-system-overhaul'.
>>
>> I've tried to implement a solution, that would clean up our
>> configure.ac/Makefiles, make Python 3 packaging possible, make
>> running tests with multiple interpreter versions possible for the
>> developers, etc.
>> I know, that I will get "crazy looks" for writing m4 macros, but I
>> really believe having stuff like 'VDSM_PY_SUPPORTED_MAJOR_VERSION' et
>> al. will make developers' lives easier.
>> I'm currently trying to address issues and comments raised by Milan,
>> I would definitely like to get others opinions on these.
>>
>> Regards, Marcin
>>
>> On 3/4/19 3:46 PM, Marcin Sobczyk wrote:
>>> On 3/1/19 2:39 PM, Milan Zamazal wrote:
>>>> Marcin Sobczyk <msobczyk(a)redhat.com> writes:
>>>>
>>>>> I've posted a series of patches:
>>>>>
https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+statu...
>>>>>
>>>>>
>>>>> They do a couple of things:
>>>>>
>>>>> * add an '--enable-python3-build' argument to
'autogen.sh'
>>>>> * modify 'Makefile.am' so it uses either
'vdsm.spec.in' or
>>>>> 'vdsm-py3.spec.in' file to build RPMs
>>>>> * preserve the 'vdsm.spec[.in]' filename in dist archives
for
>>>>> both py2
>>>>> and py3 packaging
>>>>> * fix shebang lines in a dist-hook, so that py3 packages will
>>>>> contain
>>>>> only '#!/usr/bin/python3' shebangs
>>>>> * prepare the CI for building py3 packages on fc28
>>>> Hi Marcin, cool, thank you!
>>>>
>>>>> So actually, if we replace the dummy 'vdsm-py3.spec.in' file
in
>>>>> https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the
>>>>> nasty, hacky one, or the shiny, new one), we're ready to build
py3
>>>>> packages.
>>>> I've made a dirty one: https://gerrit.ovirt.org/98150 (you can
simply
>>>> cherry pick the to wherever you need).
>>>>
>>>> However, it doesn't work, it's still built with Python 2 on my
Fedora
>>>> 28. I can see a lot of warning such as
>>>>
>>>> DEPRECATION WARNING: python2 invoked with /usr/bin/python.
>>>> Use /usr/bin/python3 or /usr/bin/python2
>>>> /usr/bin/python will be removed or switched to Python 3 in
>>>> the future.
>>>>
>>>> I can see that configure.ac in your patches still uses Python 2. Here
>>>> is what I used in my private hacks to change that. Something similar
>>>> should be done, depending on --enable-python3-build presence, and with
>>>> properly detected Python versions.
>>>
>>> Good point - I'm currently working on resolving that. It got kinda
>>> complicated but will keep you posted as soon as I get some results.
>>>
>>>>
>>>> Would you like to make the necessary changes or should I make them
>>>> myself?
>>>>
>>>> modified configure.ac
>>>> @@ -48,7 +48,7 @@ AM_INIT_AUTOMAKE([-Wno-portability])
>>>> # Checking for build tools
>>>> AC_PROG_LN_S
>>>> -AM_PATH_PYTHON([2.6])
>>>> +AM_PATH_PYTHON([3.6])
>>>> AC_ARG_ENABLE(
>>>> [hooks],
>>>> @@ -349,13 +349,13 @@ if test "$PYTHON3_SUPPORT" =
"1"; then
>>>> fi
>>>> # Checking for python-devel
>>>> -AC_PATH_PROG([PYTHON_CONFIG], [python-config])
>>>> +AC_PATH_PROG([PYTHON_CONFIG], [python3-config])
>>>> if test "x$PYTHON_CONFIG" = "x"; then
>>>> - AC_MSG_ERROR([python-devel not found, please install it.])
>>>> + AC_MSG_ERROR([python3-devel not found, please install it.])
>>>> fi
>>>> # Checking for python modules (sorted, please keep in order)
>>>> -AX_PYTHON_MODULE([six], [fatal], python2)
>>>> +AX_PYTHON_MODULE([six], [fatal], python3)
>>>> # External programs (sorted, please keep in order)
>>>> AC_PATH_PROG([CAT_PATH], [cat], [/bin/cat])
1851
days inactive
1892
days old
12 comments
5 participants
participants (5)
-
Dan Kenigsberg -
Francesco Romani -
Marcin Sobczyk -
Milan Zamazal -
Nir Soffer