Hi, On yesterday's vdsm weekly call, we were discussing the need of making Python 3 vdsm RPM packages. Some facts: - it doesn't make a lot sense to spend much time on trying to package everything - it's completely impossible i.e. to run vdsm without having 'sanlock' module - our current vdsm.spec file is crap Two non-exclusive propositions were raised: - let's try to make a quick-and-dirty patch, that will completely overwrite the existing 'vdsm.spec' (effectively making it Python 3-only) for testing purposes, and maintain it for a while - in the meantime, let's write a completely new, clean and beautiful spec file in an package-by-package, incremental manner, (also Python 3-only) that would eventually substitute the original one The quick-and-dirty spec file would be completely unsupported by CI. The new one would get a proper CI sub-stage in 'build-artifacts' stage. The steps needed to be done are: - prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds - prepare the new spec file (for now including only 'vdsm-common' package) - split 'build-artifacts' stage into 'build-py27' and 'build-py36' sub-stages (the latter currently running on fc28 only) The only package we can start with, when making the new spec file, is 'vdsm-common', as it doesn't depend on anything else (or at least I hope so...). There were also propositions about how to change the new spec file in regard to the old one (like making 'vdsm' package a meta-package). This is a good time for these propositions to be raised, reviewed and documented (something like this maybe? https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72K63...), so we can align the new spec file as we build it. I can lay the groundwork by doing the autotools/Makefiles and 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on the new spec file. Milan mentioned, that he had something like the quick-and-dirty patch, maybe he can share it with us. Questions, comments are welcome. Regards, Marcin
On Thu, Feb 28, 2019 at 1:06 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
On yesterday's vdsm weekly call, we were discussing the need of making Python 3 vdsm RPM packages.
Some facts:
- it doesn't make a lot sense to spend much time on trying to package everything - it's completely impossible i.e. to run vdsm without having 'sanlock' module
localfs storage works without sanlock, so we have a way to test the entire system without sanlock. We can make sanlock configurable to be able to build and run vdsm without it with only local storage. But I think completing the sanlock port to python 3 will be much less work.
- our current vdsm.spec file is crap
Two non-exclusive propositions were raised:
- let's try to make a quick-and-dirty patch, that will completely overwrite the existing 'vdsm.spec' (effectively making it Python 3-only) for testing purposes, and maintain it for a while - in the meantime, let's write a completely new, clean and beautiful spec file in an package-by-package, incremental manner, (also Python 3-only) that would eventually substitute the original one
The quick-and-dirty spec file would be completely unsupported by CI. The new one would get a proper CI sub-stage in 'build-artifacts' stage.
The steps needed to be done are:
- prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds - prepare the new spec file (for now including only 'vdsm-common' package) - split 'build-artifacts' stage into 'build-py27' and 'build-py36' sub-stages (the latter currently running on fc28 only)
The only package we can start with, when making the new spec file, is 'vdsm-common', as it doesn't depend on anything else (or at least I hope so...).
There were also propositions about how to change the new spec file in regard to the old one (like making 'vdsm' package a meta-package). This is a good time for these propositions to be raised, reviewed and documented (something like this maybe?
https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72K63...),
so we can align the new spec file as we build it.
I can lay the groundwork by doing the autotools/Makefiles and 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on the new spec file. Milan mentioned, that he had something like the quick-and-dirty patch, maybe he can share it with us.
Questions, comments are welcome.
Regards, Marcin
Thanks for the summary! Nir
On Thu, Feb 28, 2019 at 1:07 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
On yesterday's vdsm weekly call, we were discussing the need of making Python 3 vdsm RPM packages.
Some facts:
- it doesn't make a lot sense to spend much time on trying to package everything - it's completely impossible i.e. to run vdsm without having 'sanlock' module - our current vdsm.spec file is crap
Two non-exclusive propositions were raised:
- let's try to make a quick-and-dirty patch, that will completely overwrite the existing 'vdsm.spec' (effectively making it Python 3-only) for testing purposes, and maintain it for a while - in the meantime, let's write a completely new, clean and beautiful spec file in an package-by-package, incremental manner, (also Python 3-only) that would eventually substitute the original one
I'm not sure I understand that second option. I am afraid of fresh starts; I'd very much prefer to start from the sh*tty thing we have, and evolve it. A lot of time, re-writing a piece of software is tempting, but existing code is imbued with knowledge of past problems, which is often forgotten when you do a hard cut. Cleaning %files should be an easy first step; I think that Gal's jinja-based generation of py2/py3 packages is sane. Can you explain why not just to carry these patches over?
The quick-and-dirty spec file would be completely unsupported by CI. The new one would get a proper CI sub-stage in 'build-artifacts' stage.
The steps needed to be done are:
- prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds - prepare the new spec file (for now including only 'vdsm-common' package) - split 'build-artifacts' stage into 'build-py27' and 'build-py36' sub-stages (the latter currently running on fc28 only)
The only package we can start with, when making the new spec file, is 'vdsm-common', as it doesn't depend on anything else (or at least I hope so...).
There were also propositions about how to change the new spec file in regard to the old one (like making 'vdsm' package a meta-package). This is a good time for these propositions to be raised, reviewed and documented (something like this maybe? https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72K63...), so we can align the new spec file as we build it.
I can lay the groundwork by doing the autotools/Makefiles and 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on the new spec file. Milan mentioned, that he had something like the quick-and-dirty patch, maybe he can share it with us.
Questions, comments are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MFZHLJA46QM7PV...
Hi,
On yesterday's vdsm weekly call, we were discussing the need of making Python 3 vdsm RPM packages.
Some facts:
- it doesn't make a lot sense to spend much time on trying to package everything - it's completely impossible i.e. to run vdsm without having 'sanlock' module - our current vdsm.spec file is crap
Two non-exclusive propositions were raised:
- let's try to make a quick-and-dirty patch, that will completely overwrite the existing 'vdsm.spec' (effectively making it Python 3-only) for testing purposes, and maintain it for a while - in the meantime, let's write a completely new, clean and beautiful spec file in an package-by-package, incremental manner, (also Python 3-only) that would eventually substitute the original one I'm not sure I understand that second option. I am afraid of fresh starts; I'd very much prefer to start from the sh*tty thing we have, and evolve it. A lot of time, re-writing a piece of software is tempting, but existing code is imbued with knowledge of
On Thu, Feb 28, 2019 at 1:07 PM Marcin Sobczyk <msobczyk@redhat.com> wrote: past problems, which is often forgotten when you do a hard cut.
Cleaning %files should be an easy first step; I think that Gal's jinja-based generation of py2/py3 packages is sane. Can you explain why not just to carry these patches over? AFAIK, 4.4 will be RHEL8-only. This means, we will drop Python 2 completely. Trying to make the existing spec file work with both Python 2 and Python 3 is a wasted effort. It also complicates things a lot (new build dependency, 2 layers of
On 2/28/19 12:30 PM, Dan Kenigsberg wrote: preprocessing) unnecessarily. I'm pretty confident, that introducing the new spec file package by package, will work for us - it's not Python code, errors will probably emerge quickly. +, we would still have the original spec, until we're done, to have a reference point.
The quick-and-dirty spec file would be completely unsupported by CI. The new one would get a proper CI sub-stage in 'build-artifacts' stage.
The steps needed to be done are:
- prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds - prepare the new spec file (for now including only 'vdsm-common' package) - split 'build-artifacts' stage into 'build-py27' and 'build-py36' sub-stages (the latter currently running on fc28 only)
The only package we can start with, when making the new spec file, is 'vdsm-common', as it doesn't depend on anything else (or at least I hope so...).
There were also propositions about how to change the new spec file in regard to the old one (like making 'vdsm' package a meta-package). This is a good time for these propositions to be raised, reviewed and documented (something like this maybe? https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72K63...), so we can align the new spec file as we build it.
I can lay the groundwork by doing the autotools/Makefiles and 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on the new spec file. Milan mentioned, that he had something like the quick-and-dirty patch, maybe he can share it with us.
Questions, comments are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MFZHLJA46QM7PV...
On Thu, Feb 28, 2019 at 1:30 PM Dan Kenigsberg <danken@redhat.com> wrote:
On Thu, Feb 28, 2019 at 1:07 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
On yesterday's vdsm weekly call, we were discussing the need of making Python 3 vdsm RPM packages.
Some facts:
- it doesn't make a lot sense to spend much time on trying to package everything - it's completely impossible i.e. to run vdsm without having 'sanlock' module - our current vdsm.spec file is crap
Two non-exclusive propositions were raised:
- let's try to make a quick-and-dirty patch, that will completely overwrite the existing 'vdsm.spec' (effectively making it Python 3-only) for testing purposes, and maintain it for a while - in the meantime, let's write a completely new, clean and beautiful spec file in an package-by-package, incremental manner, (also Python 3-only) that would eventually substitute the original one
I'm not sure I understand that second option. I am afraid of fresh starts; I'd very much prefer to start from the sh*tty thing we have, and evolve it. A lot of time, re-writing a piece of software is tempting, but existing code is imbued with knowledge of past problems, which is often forgotten when you do a hard cut.
True, but if you pick a small enough component, replacing bad component with good one is much quicker. Cleaning %files should be an easy first step; This is a good idea anyway.
I think that Gal's jinja-based generation of py2/py3 packages is sane. Can you explain why not just to carry these patches over?
Adding jinna to the mix will only make the spec harder to work with, we will have spec + shell + jinna + autoconf in the same file. And when we delete the python 2 support we will have to live with that complexity that serve nothing. We need now: - python 2 for rhel 7 - python 2 for fedora - python 3 for rhel 8 - python 3 for fedora Once we have working python 3 versions, we can drop the python 2 builds, so we will have a single spec with: - pyhton 3 for rhel 8 - pyhton 3 for fedora So investing time and energy on having "smart" spec that support both python 2 and 3 is waste of effort. Idealy the new spec will be static, no code generation, no code. All code (e.g. install/upgrade scripts) should be in separate files that can be tested easily.
The quick-and-dirty spec file would be completely unsupported by CI. The new one would get a proper CI sub-stage in 'build-artifacts' stage.
The steps needed to be done are:
- prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds - prepare the new spec file (for now including only 'vdsm-common' package) - split 'build-artifacts' stage into 'build-py27' and 'build-py36' sub-stages (the latter currently running on fc28 only)
The only package we can start with, when making the new spec file, is 'vdsm-common', as it doesn't depend on anything else (or at least I hope so...).
There were also propositions about how to change the new spec file in regard to the old one (like making 'vdsm' package a meta-package). This is a good time for these propositions to be raised, reviewed and documented (something like this maybe?
https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72K63... ),
so we can align the new spec file as we build it.
I can lay the groundwork by doing the autotools/Makefiles and 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on the new spec file. Milan mentioned, that he had something like the quick-and-dirty patch, maybe he can share it with us.
Questions, comments are welcome.
Regards, Marcin
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MFZHLJA46QM7PV...
On 2/28/19 12:30 PM, Dan Kenigsberg wrote:
I am afraid of fresh starts; I'd very much prefer to start from the sh*tty thing we have, and evolve it. A lot of time, re-writing a piece of software is tempting, but existing code is imbued with knowledge of past problems, which is often forgotten when you do a hard cut.
In general, this is very true. However, in the *specific case* of the spec file, it is mostly an aggregation of fixes, hacks and workarounds arised from contingent issues. Or at very least this is my experience with the spec file. My point is the spec file is mostly a snapshot of the fixes needed for a given set of supported OSes.[1] There isn't much to salvage here, and for what's worth keeping, git history is the best record, and this is not going to be lost. For these reasons, I support Marcin's plan to start afresh side by side with a new spec file, using the old one as reference. I trust Marcin (and us) to be careful and look back often at the old spec file when writing the new one, to avoid forgetting the lessons of the past. +++ [1] the current spec file looks much more like a scratchpad full of scribbled notes than a careful curated chronicle imbued with knowledge. Bests, -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Marcin Sobczyk <msobczyk@redhat.com> writes:
Hi,
On yesterday's vdsm weekly call, we were discussing the need of making Python 3 vdsm RPM packages.
Some facts:
- it doesn't make a lot sense to spend much time on trying to package everything - it's completely impossible i.e. to run vdsm without having 'sanlock' module - our current vdsm.spec file is crap
Two non-exclusive propositions were raised:
- let's try to make a quick-and-dirty patch, that will completely overwrite the existing 'vdsm.spec' (effectively making it Python 3-only) for testing purposes, and maintain it for a while - in the meantime, let's write a completely new, clean and beautiful spec file in an package-by-package, incremental manner, (also Python 3-only) that would eventually substitute the original one
The quick-and-dirty spec file would be completely unsupported by CI. The new one would get a proper CI sub-stage in 'build-artifacts' stage.
The steps needed to be done are:
- prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds - prepare the new spec file (for now including only 'vdsm-common' package) - split 'build-artifacts' stage into 'build-py27' and 'build-py36' sub-stages (the latter currently running on fc28 only)
And solve #/usr/bin/python* issue.
The only package we can start with, when making the new spec file, is 'vdsm-common', as it doesn't depend on anything else (or at least I hope so...).
There were also propositions about how to change the new spec file in regard to the old one (like making 'vdsm' package a meta-package). This is a good time for these propositions to be raised, reviewed and documented (something like this maybe? https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72K63...), so we can align the new spec file as we build it.
I can lay the groundwork by doing the autotools/Makefiles and 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on the new spec file. Milan mentioned, that he had something like the quick-and-dirty patch, maybe he can share it with us.
I can try to post a patch with a q&d Python 3 vdsm.spec.in file; maybe later today.
Questions, comments are welcome.
Regards, Marcin
Hi, I've posted a series of patches: https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+status:m...) They do a couple of things: * add an '--enable-python3-build' argument to 'autogen.sh' * modify 'Makefile.am' so it uses either 'vdsm.spec.in' or 'vdsm-py3.spec.in' file to build RPMs * preserve the 'vdsm.spec[.in]' filename in dist archives for both py2 and py3 packaging * fix shebang lines in a dist-hook, so that py3 packages will contain only '#!/usr/bin/python3' shebangs * prepare the CI for building py3 packages on fc28 So actually, if we replace the dummy 'vdsm-py3.spec.in' file in https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the nasty, hacky one, or the shiny, new one), we're ready to build py3 packages. Reviews are more than welcome. Regards, Marcin On 2/28/19 2:02 PM, Milan Zamazal wrote:
Marcin Sobczyk <msobczyk@redhat.com> writes:
Hi,
On yesterday's vdsm weekly call, we were discussing the need of making Python 3 vdsm RPM packages.
Some facts:
- it doesn't make a lot sense to spend much time on trying to package everything - it's completely impossible i.e. to run vdsm without having 'sanlock' module - our current vdsm.spec file is crap
Two non-exclusive propositions were raised:
- let's try to make a quick-and-dirty patch, that will completely overwrite the existing 'vdsm.spec' (effectively making it Python 3-only) for testing purposes, and maintain it for a while - in the meantime, let's write a completely new, clean and beautiful spec file in an package-by-package, incremental manner, (also Python 3-only) that would eventually substitute the original one
The quick-and-dirty spec file would be completely unsupported by CI. The new one would get a proper CI sub-stage in 'build-artifacts' stage.
The steps needed to be done are:
- prepare autotools/Makefiles to differentiate Python 2/Python 3 RPM builds - prepare the new spec file (for now including only 'vdsm-common' package) - split 'build-artifacts' stage into 'build-py27' and 'build-py36' sub-stages (the latter currently running on fc28 only) And solve #/usr/bin/python* issue.
The only package we can start with, when making the new spec file, is 'vdsm-common', as it doesn't depend on anything else (or at least I hope so...).
There were also propositions about how to change the new spec file in regard to the old one (like making 'vdsm' package a meta-package). This is a good time for these propositions to be raised, reviewed and documented (something like this maybe? https://docs.google.com/document/d/13EXN1Iwq-OPoc2A5Y3PJBpOiNC10ugx6eCE72K63...), so we can align the new spec file as we build it.
I can lay the groundwork by doing the autotools/Makefiles and 'build-artifacts' splitting. Gal Zaidman agreed on starting to work on the new spec file. Milan mentioned, that he had something like the quick-and-dirty patch, maybe he can share it with us. I can try to post a patch with a q&d Python 3 vdsm.spec.in file; maybe later today.
Questions, comments are welcome.
Regards, Marcin
Marcin Sobczyk <msobczyk@redhat.com> writes:
I've posted a series of patches: https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+status:m...)
They do a couple of things:
* add an '--enable-python3-build' argument to 'autogen.sh' * modify 'Makefile.am' so it uses either 'vdsm.spec.in' or 'vdsm-py3.spec.in' file to build RPMs * preserve the 'vdsm.spec[.in]' filename in dist archives for both py2 and py3 packaging * fix shebang lines in a dist-hook, so that py3 packages will contain only '#!/usr/bin/python3' shebangs * prepare the CI for building py3 packages on fc28
Hi Marcin, cool, thank you!
So actually, if we replace the dummy 'vdsm-py3.spec.in' file in https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the nasty, hacky one, or the shiny, new one), we're ready to build py3 packages.
I've made a dirty one: https://gerrit.ovirt.org/98150 (you can simply cherry pick the to wherever you need). However, it doesn't work, it's still built with Python 2 on my Fedora 28. I can see a lot of warning such as DEPRECATION WARNING: python2 invoked with /usr/bin/python. Use /usr/bin/python3 or /usr/bin/python2 /usr/bin/python will be removed or switched to Python 3 in the future. I can see that configure.ac in your patches still uses Python 2. Here is what I used in my private hacks to change that. Something similar should be done, depending on --enable-python3-build presence, and with properly detected Python versions. Would you like to make the necessary changes or should I make them myself? modified configure.ac @@ -48,7 +48,7 @@ AM_INIT_AUTOMAKE([-Wno-portability]) # Checking for build tools AC_PROG_LN_S -AM_PATH_PYTHON([2.6]) +AM_PATH_PYTHON([3.6]) AC_ARG_ENABLE( [hooks], @@ -349,13 +349,13 @@ if test "$PYTHON3_SUPPORT" = "1"; then fi # Checking for python-devel -AC_PATH_PROG([PYTHON_CONFIG], [python-config]) +AC_PATH_PROG([PYTHON_CONFIG], [python3-config]) if test "x$PYTHON_CONFIG" = "x"; then - AC_MSG_ERROR([python-devel not found, please install it.]) + AC_MSG_ERROR([python3-devel not found, please install it.]) fi # Checking for python modules (sorted, please keep in order) -AX_PYTHON_MODULE([six], [fatal], python2) +AX_PYTHON_MODULE([six], [fatal], python3) # External programs (sorted, please keep in order) AC_PATH_PROG([CAT_PATH], [cat], [/bin/cat])
On 3/1/19 2:39 PM, Milan Zamazal wrote:
Marcin Sobczyk <msobczyk@redhat.com> writes:
I've posted a series of patches: https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+status:m...)
They do a couple of things:
* add an '--enable-python3-build' argument to 'autogen.sh' * modify 'Makefile.am' so it uses either 'vdsm.spec.in' or 'vdsm-py3.spec.in' file to build RPMs * preserve the 'vdsm.spec[.in]' filename in dist archives for both py2 and py3 packaging * fix shebang lines in a dist-hook, so that py3 packages will contain only '#!/usr/bin/python3' shebangs * prepare the CI for building py3 packages on fc28 Hi Marcin, cool, thank you!
So actually, if we replace the dummy 'vdsm-py3.spec.in' file in https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the nasty, hacky one, or the shiny, new one), we're ready to build py3 packages. I've made a dirty one: https://gerrit.ovirt.org/98150 (you can simply cherry pick the to wherever you need).
However, it doesn't work, it's still built with Python 2 on my Fedora 28. I can see a lot of warning such as
DEPRECATION WARNING: python2 invoked with /usr/bin/python. Use /usr/bin/python3 or /usr/bin/python2 /usr/bin/python will be removed or switched to Python 3 in the future.
I can see that configure.ac in your patches still uses Python 2. Here is what I used in my private hacks to change that. Something similar should be done, depending on --enable-python3-build presence, and with properly detected Python versions.
Good point - I'm currently working on resolving that. It got kinda complicated but will keep you posted as soon as I get some results.
Would you like to make the necessary changes or should I make them myself?
modified configure.ac @@ -48,7 +48,7 @@ AM_INIT_AUTOMAKE([-Wno-portability])
# Checking for build tools AC_PROG_LN_S -AM_PATH_PYTHON([2.6]) +AM_PATH_PYTHON([3.6])
AC_ARG_ENABLE( [hooks], @@ -349,13 +349,13 @@ if test "$PYTHON3_SUPPORT" = "1"; then fi
# Checking for python-devel -AC_PATH_PROG([PYTHON_CONFIG], [python-config]) +AC_PATH_PROG([PYTHON_CONFIG], [python3-config]) if test "x$PYTHON_CONFIG" = "x"; then - AC_MSG_ERROR([python-devel not found, please install it.]) + AC_MSG_ERROR([python3-devel not found, please install it.]) fi
# Checking for python modules (sorted, please keep in order) -AX_PYTHON_MODULE([six], [fatal], python2) +AX_PYTHON_MODULE([six], [fatal], python3)
# External programs (sorted, please keep in order) AC_PATH_PROG([CAT_PATH], [cat], [/bin/cat])
Hi, it took a while... after trying to grasp "the big picture", I came up with these: https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:bu... The old topic: https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:py... was rebased (not yet 100%) on top of 'build-system-overhaul'. I've tried to implement a solution, that would clean up our configure.ac/Makefiles, make Python 3 packaging possible, make running tests with multiple interpreter versions possible for the developers, etc. I know, that I will get "crazy looks" for writing m4 macros, but I really believe having stuff like 'VDSM_PY_SUPPORTED_MAJOR_VERSION' et al. will make developers' lives easier. I'm currently trying to address issues and comments raised by Milan, I would definitely like to get others opinions on these. Regards, Marcin On 3/4/19 3:46 PM, Marcin Sobczyk wrote:
On 3/1/19 2:39 PM, Milan Zamazal wrote:
Marcin Sobczyk <msobczyk@redhat.com> writes:
I've posted a series of patches: https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+status:m...)
They do a couple of things:
* add an '--enable-python3-build' argument to 'autogen.sh' * modify 'Makefile.am' so it uses either 'vdsm.spec.in' or 'vdsm-py3.spec.in' file to build RPMs * preserve the 'vdsm.spec[.in]' filename in dist archives for both py2 and py3 packaging * fix shebang lines in a dist-hook, so that py3 packages will contain only '#!/usr/bin/python3' shebangs * prepare the CI for building py3 packages on fc28 Hi Marcin, cool, thank you!
So actually, if we replace the dummy 'vdsm-py3.spec.in' file in https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the nasty, hacky one, or the shiny, new one), we're ready to build py3 packages. I've made a dirty one: https://gerrit.ovirt.org/98150 (you can simply cherry pick the to wherever you need).
However, it doesn't work, it's still built with Python 2 on my Fedora 28. I can see a lot of warning such as
DEPRECATION WARNING: python2 invoked with /usr/bin/python. Use /usr/bin/python3 or /usr/bin/python2 /usr/bin/python will be removed or switched to Python 3 in the future.
I can see that configure.ac in your patches still uses Python 2. Here is what I used in my private hacks to change that. Something similar should be done, depending on --enable-python3-build presence, and with properly detected Python versions.
Good point - I'm currently working on resolving that. It got kinda complicated but will keep you posted as soon as I get some results.
Would you like to make the necessary changes or should I make them myself?
modified configure.ac @@ -48,7 +48,7 @@ AM_INIT_AUTOMAKE([-Wno-portability]) # Checking for build tools AC_PROG_LN_S -AM_PATH_PYTHON([2.6]) +AM_PATH_PYTHON([3.6]) AC_ARG_ENABLE( [hooks], @@ -349,13 +349,13 @@ if test "$PYTHON3_SUPPORT" = "1"; then fi # Checking for python-devel -AC_PATH_PROG([PYTHON_CONFIG], [python-config]) +AC_PATH_PROG([PYTHON_CONFIG], [python3-config]) if test "x$PYTHON_CONFIG" = "x"; then - AC_MSG_ERROR([python-devel not found, please install it.]) + AC_MSG_ERROR([python3-devel not found, please install it.]) fi # Checking for python modules (sorted, please keep in order) -AX_PYTHON_MODULE([six], [fatal], python2) +AX_PYTHON_MODULE([six], [fatal], python3) # External programs (sorted, please keep in order) AC_PATH_PROG([CAT_PATH], [cat], [/bin/cat])
Hi, I've made a google document, that describes the changes that come with the patches and the rationale behind them: https://docs.google.com/document/d/1XBf9eP0lh3IeRzkQ3X-c191rMpOwsTTkX-eaoPqW... Please note, that the patches do not reflect 100% what's described in the doc - while writing the doc I came to conclusions, that some things should be improved. Since Milan told, that Python 3 packaging is very high priority for him (even if the packages don't actually work yet), I would like to ask you to get involved in reviewing the document. After we establish a common ground, I will align my patches to reflect what's in the doc. P.S. Previous Markdown version was a pain to use so I just rewrote the thing as a simple doc. Thanks, Marcin On 3/18/19 12:19 PM, Marcin Sobczyk wrote:
Hi,
it took a while... after trying to grasp "the big picture", I came up with these:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:bu...
The old topic:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:py...
was rebased (not yet 100%) on top of 'build-system-overhaul'.
I've tried to implement a solution, that would clean up our configure.ac/Makefiles, make Python 3 packaging possible, make running tests with multiple interpreter versions possible for the developers, etc. I know, that I will get "crazy looks" for writing m4 macros, but I really believe having stuff like 'VDSM_PY_SUPPORTED_MAJOR_VERSION' et al. will make developers' lives easier. I'm currently trying to address issues and comments raised by Milan, I would definitely like to get others opinions on these.
Regards, Marcin
On 3/4/19 3:46 PM, Marcin Sobczyk wrote:
On 3/1/19 2:39 PM, Milan Zamazal wrote:
Marcin Sobczyk <msobczyk@redhat.com> writes:
I've posted a series of patches: https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+status:m...)
They do a couple of things:
* add an '--enable-python3-build' argument to 'autogen.sh' * modify 'Makefile.am' so it uses either 'vdsm.spec.in' or 'vdsm-py3.spec.in' file to build RPMs * preserve the 'vdsm.spec[.in]' filename in dist archives for both py2 and py3 packaging * fix shebang lines in a dist-hook, so that py3 packages will contain only '#!/usr/bin/python3' shebangs * prepare the CI for building py3 packages on fc28 Hi Marcin, cool, thank you!
So actually, if we replace the dummy 'vdsm-py3.spec.in' file in https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the nasty, hacky one, or the shiny, new one), we're ready to build py3 packages. I've made a dirty one: https://gerrit.ovirt.org/98150 (you can simply cherry pick the to wherever you need).
However, it doesn't work, it's still built with Python 2 on my Fedora 28. I can see a lot of warning such as
DEPRECATION WARNING: python2 invoked with /usr/bin/python. Use /usr/bin/python3 or /usr/bin/python2 /usr/bin/python will be removed or switched to Python 3 in the future.
I can see that configure.ac in your patches still uses Python 2. Here is what I used in my private hacks to change that. Something similar should be done, depending on --enable-python3-build presence, and with properly detected Python versions.
Good point - I'm currently working on resolving that. It got kinda complicated but will keep you posted as soon as I get some results.
Would you like to make the necessary changes or should I make them myself?
modified configure.ac @@ -48,7 +48,7 @@ AM_INIT_AUTOMAKE([-Wno-portability]) # Checking for build tools AC_PROG_LN_S -AM_PATH_PYTHON([2.6]) +AM_PATH_PYTHON([3.6]) AC_ARG_ENABLE( [hooks], @@ -349,13 +349,13 @@ if test "$PYTHON3_SUPPORT" = "1"; then fi # Checking for python-devel -AC_PATH_PROG([PYTHON_CONFIG], [python-config]) +AC_PATH_PROG([PYTHON_CONFIG], [python3-config]) if test "x$PYTHON_CONFIG" = "x"; then - AC_MSG_ERROR([python-devel not found, please install it.]) + AC_MSG_ERROR([python3-devel not found, please install it.]) fi # Checking for python modules (sorted, please keep in order) -AX_PYTHON_MODULE([six], [fatal], python2) +AX_PYTHON_MODULE([six], [fatal], python3) # External programs (sorted, please keep in order) AC_PATH_PROG([CAT_PATH], [cat], [/bin/cat])
Hi, a quick follow up on my py3-packaging patches. I've split them into parts, because there's a lot of them, and some of the older patches needed to be rebased on top of new stuff. Two issues arised when working on the patches. The first one is trying to support both py36 and py37 on Fedora 29. We use 'automation/*packages*' files to decide what needs to be installed in order to run the tests. These files contain names of yum/dnf packages. The thing is, that these packages are installed into '/usr/lib/python3.7/site-packages'. When trying to run py36 tests, they can't be found. This can be fixed, by including the test dependencies in 'tox.ini' in the 'deps' section. This is how tox handles dependencies: - if the package is already available in the OS, it uses that version (py37 case) - if the package is not available, it installs it to its local environment with 'pip' (py36 case) Listing the dependencies in the 'deps' section fixes the problem... with one exception. 'selinux' python bindings are not available in PyPI - there's only the yum/dnf package version. A workaround would be to mark the tests that require 'selinux' with something like 'test_on_target_python_only'. I think we should focus on the packaging stuff right now, so I would postpone the fix for this problem. The first batch of patches [1] partially deals with the issue by a workaround. It limits the tested interpreter versions to "python2.7 python3" instead of "python2.7 python3.6 python3.7". The effect is, that if a developer runs Fedora 29, and has both py36 and py37 installed, only py37 will be used by default (that's the one that 'python3' is pointing to). CI is unaffected by this change (since there's only one py3 version anyway). The second issue is byte-compiling our python modules. Everything suggests that we're doing it twice (once by 'build-aux/py-compile' and the second time is done by 'brp-python-bytecompile' script [2]) as there are files like 'API.cpython-37.pyc' and 'API.cpython-37.opt-1.pyc' in the RPM build root dir, and they are identical. I'm still working on cleaning this mess up. Other than that, there are patches that unify the makefile's test targets [3] and the ones that add autogen.sh flags as described in the design document [4], which are ready to be reviewed. It would be nice if someone else played around a bit with the flags - I use them all the time and they seem to work for me, but I'm already occupied with the byte-compiling issue. Thanks, Marcin [1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:bu... [2] https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#m... [3] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:bu... [4] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:bu... On 3/28/19 11:48 AM, Marcin Sobczyk wrote:
Hi,
I've made a google document, that describes the changes that come with the patches and the rationale behind them:
https://docs.google.com/document/d/1XBf9eP0lh3IeRzkQ3X-c191rMpOwsTTkX-eaoPqW...
Please note, that the patches do not reflect 100% what's described in the doc - while writing the doc I came to conclusions, that some things should be improved. Since Milan told, that Python 3 packaging is very high priority for him (even if the packages don't actually work yet), I would like to ask you to get involved in reviewing the document. After we establish a common ground, I will align my patches to reflect what's in the doc.
P.S. Previous Markdown version was a pain to use so I just rewrote the thing as a simple doc.
Thanks, Marcin
On 3/18/19 12:19 PM, Marcin Sobczyk wrote:
Hi,
it took a while... after trying to grasp "the big picture", I came up with these:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:bu...
The old topic:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:py...
was rebased (not yet 100%) on top of 'build-system-overhaul'.
I've tried to implement a solution, that would clean up our configure.ac/Makefiles, make Python 3 packaging possible, make running tests with multiple interpreter versions possible for the developers, etc. I know, that I will get "crazy looks" for writing m4 macros, but I really believe having stuff like 'VDSM_PY_SUPPORTED_MAJOR_VERSION' et al. will make developers' lives easier. I'm currently trying to address issues and comments raised by Milan, I would definitely like to get others opinions on these.
Regards, Marcin
On 3/4/19 3:46 PM, Marcin Sobczyk wrote:
On 3/1/19 2:39 PM, Milan Zamazal wrote:
Marcin Sobczyk <msobczyk@redhat.com> writes:
I've posted a series of patches: https://gerrit.ovirt.org/#/q/topic:py3-rpm-packages+(status:open+OR+status:m...)
They do a couple of things:
* add an '--enable-python3-build' argument to 'autogen.sh' * modify 'Makefile.am' so it uses either 'vdsm.spec.in' or 'vdsm-py3.spec.in' file to build RPMs * preserve the 'vdsm.spec[.in]' filename in dist archives for both py2 and py3 packaging * fix shebang lines in a dist-hook, so that py3 packages will contain only '#!/usr/bin/python3' shebangs * prepare the CI for building py3 packages on fc28 Hi Marcin, cool, thank you!
So actually, if we replace the dummy 'vdsm-py3.spec.in' file in https://gerrit.ovirt.org/#/c/98119/ with something useful (be it the nasty, hacky one, or the shiny, new one), we're ready to build py3 packages. I've made a dirty one: https://gerrit.ovirt.org/98150 (you can simply cherry pick the to wherever you need).
However, it doesn't work, it's still built with Python 2 on my Fedora 28. I can see a lot of warning such as
DEPRECATION WARNING: python2 invoked with /usr/bin/python. Use /usr/bin/python3 or /usr/bin/python2 /usr/bin/python will be removed or switched to Python 3 in the future.
I can see that configure.ac in your patches still uses Python 2. Here is what I used in my private hacks to change that. Something similar should be done, depending on --enable-python3-build presence, and with properly detected Python versions.
Good point - I'm currently working on resolving that. It got kinda complicated but will keep you posted as soon as I get some results.
Would you like to make the necessary changes or should I make them myself?
modified configure.ac @@ -48,7 +48,7 @@ AM_INIT_AUTOMAKE([-Wno-portability]) # Checking for build tools AC_PROG_LN_S -AM_PATH_PYTHON([2.6]) +AM_PATH_PYTHON([3.6]) AC_ARG_ENABLE( [hooks], @@ -349,13 +349,13 @@ if test "$PYTHON3_SUPPORT" = "1"; then fi # Checking for python-devel -AC_PATH_PROG([PYTHON_CONFIG], [python-config]) +AC_PATH_PROG([PYTHON_CONFIG], [python3-config]) if test "x$PYTHON_CONFIG" = "x"; then - AC_MSG_ERROR([python-devel not found, please install it.]) + AC_MSG_ERROR([python3-devel not found, please install it.]) fi # Checking for python modules (sorted, please keep in order) -AX_PYTHON_MODULE([six], [fatal], python2) +AX_PYTHON_MODULE([six], [fatal], python3) # External programs (sorted, please keep in order) AC_PATH_PROG([CAT_PATH], [cat], [/bin/cat])
participants (5)
-
Dan Kenigsberg -
Francesco Romani -
Marcin Sobczyk -
Milan Zamazal -
Nir Soffer