Hi, there have been efforts by different Vdsm developers to port Vdsm to Python 3. I've joined the efforts recently and I focused on the aim to make Vdsm running (not necessarily working correctly initially) on Python 3, in order to better understand all the related problems and to test Python 3 compatibility (among other) on a running Vdsm instance. I can currently start Vdsm on Python 3, although it can't communicate with clients yet (such as Engine), see below. FWIW, here is what I had to do to make Vdsm at least installable and startable with Python 3. I use Fedora 28 as the platform. The primary reason is that ovirt.org provides for Fedora 28 all the non-Vdsm oVirt packages needed to run Vdsm on the host. Direct host installation doesn't work for me yet with Python 3. It might be a nice next goal but there is an added complexity in requiring proper support in all the related oVirt packages, not just Vdsm. It's better to have a basically working Vdsm first. So I had installed the host with Python 2 Vdsm and then upgraded Vdsm to Python 3 version. One non-Vdsm Python package required by Vdsm and not yet ported to Python 3 is sanlock-python. There is an ongoing work on Python 3 port at https://pagure.io/fork/mcsontos/sanlock/commits/dev-mcsontos-python3. I didn't know about it so I used my own hack that made sanlock Python module at least importable on Python 3. Let's discuss it on sanlock-devel. Regarding Vdsm itself, there are several areas to fix. First, there are compatibility problems in Python code. I started making some fixes in a patch series starting with https://gerrit.ovirt.org/97019. Please note there are other Python 3 patches by other Vdsm developers. I hope that once we branch out 4.3, we can start merging the patches to master quickly and better coordinate our efforts. Then there are other changes to Vdsm needed, which I don't have patches for (just my ugly private hacks): - All the /usr/bin/python2 shebangs must be replaced with /usr/bin/python3. We should find a good way to solve and automated that, perhaps as part of the installation process depending on Python version selected. - configure.ac must be adjusted: + AM_PATH_PYTHON([3.6]) + AC_PATH_PROG([PYTHON_CONFIG], [python3-config]) + AX_PYTHON_MODULE([six], [fatal], python3) It should be easy to make a patch for that. - The following changes in vdsm.spec.in are needed: + Dependencies on Python packages must be adjusted. + python3_sitelib must be used instead of python_sitelib for Python 3. + Byte compiled files are placed into __pycache__ subdirectories by Python 3 and they must be installed. It would be nice to have patches for that so that we can build Python 3 rpm's easily. That's it at the moment. Most compatibility problems in Python code are easy to fix once they are identified. One important exception is bytes versus strings -- that's going to be delicate. The bytes versus strings problem is very present in communication code -- such as vdsm/rpc/http.py, vdsm/sslutils.py, etc. While strings constants are typically used in the code, the communication means work with bytes. This mismatch prevents Vdsm from communicating with the clients, so it would be very useful if someone could look at it and fix the code (one of the harder parts would probably be to decide where to keep bytes and where to already convert them to strings). HTH. As more people are involved in porting Vdsm to Python 3, it might be useful to coordinate all the efforts here, to avoid duplicate work, to share findings and to speed up the process. Thanks, Milan
On Thu, Feb 7, 2019 at 11:04 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Hi, there have been efforts by different Vdsm developers to port Vdsm to Python 3. I've joined the efforts recently and I focused on the aim to make Vdsm running (not necessarily working correctly initially) on Python 3, in order to better understand all the related problems and to test Python 3 compatibility (among other) on a running Vdsm instance.
I can currently start Vdsm on Python 3, although it can't communicate with clients yet (such as Engine), see below. FWIW, here is what I had to do to make Vdsm at least installable and startable with Python 3.
I use Fedora 28 as the platform. The primary reason is that ovirt.org provides for Fedora 28 all the non-Vdsm oVirt packages needed to run Vdsm on the host.
Direct host installation doesn't work for me yet with Python 3. It might be a nice next goal but there is an added complexity in requiring proper support in all the related oVirt packages, not just Vdsm. It's better to have a basically working Vdsm first. So I had installed the host with Python 2 Vdsm and then upgraded Vdsm to Python 3 version.
One non-Vdsm Python package required by Vdsm and not yet ported to Python 3 is sanlock-python. There is an ongoing work on Python 3 port at https://pagure.io/fork/mcsontos/sanlock/commits/dev-mcsontos-python3. I didn't know about it so I used my own hack that made sanlock Python module at least importable on Python 3. Let's discuss it on sanlock-devel.
Regarding Vdsm itself, there are several areas to fix. First, there are compatibility problems in Python code. I started making some fixes in a patch series starting with https://gerrit.ovirt.org/97019. Please note there are other Python 3 patches by other Vdsm developers. I hope that once we branch out 4.3, we can start merging the patches to master quickly and better coordinate our efforts.
Then there are other changes to Vdsm needed, which I don't have patches for (just my ugly private hacks):
I believe that +Gal Zaidman has few patches for these in his drawer. He has postponed his work due to 4.3 stabilitzation.
- All the /usr/bin/python2 shebangs must be replaced with /usr/bin/python3. We should find a good way to solve and automated that, perhaps as part of the installation process depending on Python version selected.
- configure.ac must be adjusted:
+ AM_PATH_PYTHON([3.6]) + AC_PATH_PROG([PYTHON_CONFIG], [python3-config]) + AX_PYTHON_MODULE([six], [fatal], python3)
It should be easy to make a patch for that.
- The following changes in vdsm.spec.in are needed:
+ Dependencies on Python packages must be adjusted. + python3_sitelib must be used instead of python_sitelib for Python 3. + Byte compiled files are placed into __pycache__ subdirectories by Python 3 and they must be installed.
It would be nice to have patches for that so that we can build Python 3 rpm's easily.
That's it at the moment. Most compatibility problems in Python code are easy to fix once they are identified. One important exception is bytes versus strings -- that's going to be delicate.
The bytes versus strings problem is very present in communication code -- such as vdsm/rpc/http.py, vdsm/sslutils.py, etc. While strings constants are typically used in the code, the communication means work with bytes. This mismatch prevents Vdsm from communicating with the clients, so it would be very useful if someone could look at it and fix the code (one of the harder parts would probably be to decide where to keep bytes and where to already convert them to strings).
HTH. As more people are involved in porting Vdsm to Python 3, it might be useful to coordinate all the efforts here, to avoid duplicate work, to share findings and to speed up the process.
Indeed. We may also try to revive Trello usage. Would you add relevant tasks to https://trello.com/b/tHeO1BFW/py3-vdsm ?
Thanks, Milan _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MRGBB2VJJX6DTU...
Dan Kenigsberg <danken@redhat.com> writes:
On Thu, Feb 7, 2019 at 11:04 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Hi, there have been efforts by different Vdsm developers to port Vdsm to
Python 3. I've joined the efforts recently and I focused on the aim to make Vdsm running (not necessarily working correctly initially) on Python 3, in order to better understand all the related problems and to test Python 3 compatibility (among other) on a running Vdsm instance.
I can currently start Vdsm on Python 3, although it can't communicate with clients yet (such as Engine), see below. FWIW, here is what I had to do to make Vdsm at least installable and startable with Python 3.
I use Fedora 28 as the platform. The primary reason is that ovirt.org provides for Fedora 28 all the non-Vdsm oVirt packages needed to run Vdsm on the host.
Direct host installation doesn't work for me yet with Python 3. It might be a nice next goal but there is an added complexity in requiring proper support in all the related oVirt packages, not just Vdsm. It's better to have a basically working Vdsm first. So I had installed the host with Python 2 Vdsm and then upgraded Vdsm to Python 3 version.
One non-Vdsm Python package required by Vdsm and not yet ported to Python 3 is sanlock-python. There is an ongoing work on Python 3 port at https://pagure.io/fork/mcsontos/sanlock/commits/dev-mcsontos-python3. I didn't know about it so I used my own hack that made sanlock Python module at least importable on Python 3. Let's discuss it on sanlock-devel.
Regarding Vdsm itself, there are several areas to fix. First, there are compatibility problems in Python code. I started making some fixes in a patch series starting with https://gerrit.ovirt.org/97019. Please note there are other Python 3 patches by other Vdsm developers. I hope that once we branch out 4.3, we can start merging the patches to master quickly and better coordinate our efforts.
Then there are other changes to Vdsm needed, which I don't have patches for (just my ugly private hacks):
I believe that +Gal Zaidman has few patches for these in his drawer. He has postponed his work due to 4.3 stabilitzation.
Yes. Gal, perhaps you can work on making the patches ready in the meantime, so that they can be merged quickly once we make 4.3 branch?
- All the /usr/bin/python2 shebangs must be replaced with /usr/bin/python3. We should find a good way to solve and automated that, perhaps as part of the installation process depending on Python version selected.
- configure.ac must be adjusted:
+ AM_PATH_PYTHON([3.6]) + AC_PATH_PROG([PYTHON_CONFIG], [python3-config]) + AX_PYTHON_MODULE([six], [fatal], python3)
It should be easy to make a patch for that.
- The following changes in vdsm.spec.in are needed:
+ Dependencies on Python packages must be adjusted. + python3_sitelib must be used instead of python_sitelib for Python 3. + Byte compiled files are placed into __pycache__ subdirectories by Python 3 and they must be installed.
It would be nice to have patches for that so that we can build Python 3 rpm's easily.
That's it at the moment. Most compatibility problems in Python code are easy to fix once they are identified. One important exception is bytes versus strings -- that's going to be delicate.
The bytes versus strings problem is very present in communication code -- such as vdsm/rpc/http.py, vdsm/sslutils.py, etc. While strings constants are typically used in the code, the communication means work with bytes. This mismatch prevents Vdsm from communicating with the clients, so it would be very useful if someone could look at it and fix the code (one of the harder parts would probably be to decide where to keep bytes and where to already convert them to strings).
HTH. As more people are involved in porting Vdsm to Python 3, it might be useful to coordinate all the efforts here, to avoid duplicate work, to share findings and to speed up the process.
Indeed. We may also try to revive Trello usage. Would you add relevant tasks to https://trello.com/b/tHeO1BFW/py3-vdsm ?
Sure, a couple of tasks added.
On Fri, Feb 8, 2019 at 10:42 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Dan Kenigsberg <danken@redhat.com> writes:
On Thu, Feb 7, 2019 at 11:04 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Hi, there have been efforts by different Vdsm developers to port Vdsm to
Python 3. I've joined the efforts recently and I focused on the aim to make Vdsm running (not necessarily working correctly initially) on Python 3, in order to better understand all the related problems and to test Python 3 compatibility (among other) on a running Vdsm instance.
I can currently start Vdsm on Python 3, although it can't communicate with clients yet (such as Engine), see below. FWIW, here is what I had to do to make Vdsm at least installable and startable with Python 3.
I use Fedora 28 as the platform. The primary reason is that ovirt.org provides for Fedora 28 all the non-Vdsm oVirt packages needed to run Vdsm on the host.
Direct host installation doesn't work for me yet with Python 3. It might be a nice next goal but there is an added complexity in requiring proper support in all the related oVirt packages, not just Vdsm. It's better to have a basically working Vdsm first. So I had installed the host with Python 2 Vdsm and then upgraded Vdsm to Python 3 version.
I remember that I once tried installing the engine with python3, and adding an fc28 host worked fine, what failed when you tried it ?
One non-Vdsm Python package required by Vdsm and not yet ported to Python 3 is sanlock-python. There is an ongoing work on Python 3 port at https://pagure.io/fork/mcsontos/sanlock/commits/dev-mcsontos-python3 . I didn't know about it so I used my own hack that made sanlock Python module at least importable on Python 3. Let's discuss it on sanlock-devel.
Regarding Vdsm itself, there are several areas to fix. First, there are compatibility problems in Python code. I started making some fixes in a patch series starting with https://gerrit.ovirt.org/97019. Please note there are other Python 3 patches by other Vdsm developers. I hope that once we branch out 4.3, we can start merging the patches to master quickly and better coordinate our efforts.
Then there are other changes to Vdsm needed, which I don't have patches for (just my ugly private hacks):
I believe that +Gal Zaidman has few patches for these in his drawer. He has postponed his work due to 4.3 stabilitzation.
Yes. Gal, perhaps you can work on making the patches ready in the meantime, so that they can be merged quickly once we make 4.3 branch?
I currently understand that in terms of packaging we have to do some stuff before we can add the python2/3 packages in VDSM: 1. Clean the spec %files section: I send a patch: https://gerrit.ovirt.org/#/c/97266/ which starts the cleaning process, I had some questions there to see if we like the direction of this patch, would be happy to get your opinion on it :) 2. Agree on a way to package without duplicating code, I had this patch https://gerrit.ovirt.org/#/c/96996/ which started the discussion, I think that a good approach is to use Jinja templeting as a layer on top of the spec.in instead of the script that I added, maybe even have a python-vdsm-packages.in file which we can use to generate the %packages, I will work on a patch for that soon.
- All the /usr/bin/python2 shebangs must be replaced with /usr/bin/python3. We should find a good way to solve and automated that, perhaps as part of the installation process depending on Python version selected.
- configure.ac must be adjusted:
+ AM_PATH_PYTHON([3.6]) + AC_PATH_PROG([PYTHON_CONFIG], [python3-config]) + AX_PYTHON_MODULE([six], [fatal], python3)
It should be easy to make a patch for that.
I think that we should remove most of the python2/3 stuff from the configure.ac such as PYTHON3_SUPPORT, fedora_run and all those stuff and decide which python version we are using in the spec file based on the OS and pass it as a parameter to the configure.
- The following changes in vdsm.spec.in are needed:
+ Dependencies on Python packages must be adjusted. + python3_sitelib must be used instead of python_sitelib for Python 3. + Byte compiled files are placed into __pycache__ subdirectories by Python 3 and they must be installed.
we will do all that per package once we will start packaging
It would be nice to have patches for that so that we can build Python 3 rpm's easily.
That's it at the moment. Most compatibility problems in Python code are easy to fix once they are identified. One important exception is bytes versus strings -- that's going to be delicate.
The bytes versus strings problem is very present in communication code -- such as vdsm/rpc/http.py, vdsm/sslutils.py, etc. While strings constants are typically used in the code, the communication means work with bytes. This mismatch prevents Vdsm from communicating with the clients, so it would be very useful if someone could look at it and fix the code (one of the harder parts would probably be to decide where to keep bytes and where to already convert them to strings).
HTH. As more people are involved in porting Vdsm to Python 3, it might be useful to coordinate all the efforts here, to avoid duplicate work, to share findings and to speed up the process.
Indeed. We may also try to revive Trello usage. Would you add relevant tasks to https://trello.com/b/tHeO1BFW/py3-vdsm ?
Sure, a couple of tasks added.
Hi, On 2/10/19 10:41 AM, Gal Zaidman wrote:
On Fri, Feb 8, 2019 at 10:42 PM Milan Zamazal <mzamazal@redhat.com <mailto:mzamazal@redhat.com>> wrote:
Dan Kenigsberg <danken@redhat.com <mailto:danken@redhat.com>> writes:
> On Thu, Feb 7, 2019 at 11:04 PM Milan Zamazal <mzamazal@redhat.com <mailto:mzamazal@redhat.com>> wrote: >> >> Hi, there have been efforts by different Vdsm developers to port Vdsm to > >> Python 3. I've joined the efforts recently and I focused on the aim to >> make Vdsm running (not necessarily working correctly initially) on >> Python 3, in order to better understand all the related problems and to >> test Python 3 compatibility (among other) on a running Vdsm instance. >> >> I can currently start Vdsm on Python 3, although it can't communicate >> with clients yet (such as Engine), see below. FWIW, here is what I had >> to do to make Vdsm at least installable and startable with Python 3. >> >> I use Fedora 28 as the platform. The primary reason is that ovirt.org <http://ovirt.org> >> provides for Fedora 28 all the non-Vdsm oVirt packages needed to run >> Vdsm on the host. >> >> Direct host installation doesn't work for me yet with Python 3. It >> might be a nice next goal but there is an added complexity in requiring >> proper support in all the related oVirt packages, not just Vdsm. It's >> better to have a basically working Vdsm first. So I had installed the >> host with Python 2 Vdsm and then upgraded Vdsm to Python 3 version.
I remember that I once tried installing the engine with python3, and adding an fc28 host worked fine, what failed when you tried it ?
>> One non-Vdsm Python package required by Vdsm and not yet ported to >> Python 3 is sanlock-python. There is an ongoing work on Python 3 port >> at https://pagure.io/fork/mcsontos/sanlock/commits/dev-mcsontos-python3. >> I didn't know about it so I used my own hack that made sanlock Python >> module at least importable on Python 3. Let's discuss it on >> sanlock-devel. >> >> Regarding Vdsm itself, there are several areas to fix. First, there are >> compatibility problems in Python code. I started making some fixes in a >> patch series starting with https://gerrit.ovirt.org/97019. Please note >> there are other Python 3 patches by other Vdsm developers. I hope that >> once we branch out 4.3, we can start merging the patches to master >> quickly and better coordinate our efforts. >> >> Then there are other changes to Vdsm needed, which I don't have patches >> for (just my ugly private hacks): > > I believe that +Gal Zaidman has few patches for these in his drawer. > He has postponed his work due to 4.3 stabilitzation.
Yes. Gal, perhaps you can work on making the patches ready in the meantime, so that they can be merged quickly once we make 4.3 branch?
I currently understand that in terms of packaging we have to do some stuff before we can add the python2/3 packages in VDSM: 1. Clean the spec %files section: I send a patch: https://gerrit.ovirt.org/#/c/97266/ which starts the cleaning process, I had some questions there to see if we like the direction of this patch, would be happy to get your opinion on it :) 2. Agree on a way to package without duplicating code, I had this patch https://gerrit.ovirt.org/#/c/96996/ which started the discussion, I think that a good approach is to use Jinja templeting as a layer on top of the spec.in <http://spec.in> instead of the script that I added, maybe even have a python-vdsm-packages.in <http://python-vdsm-packages.in> file which we can use to generate the %packages, I will work on a patch for that soon.
>> - All the /usr/bin/python2 shebangs must be replaced with >> /usr/bin/python3. We should find a good way to solve and automated >> that, perhaps as part of the installation process depending on Python >> version selected.
Regarding python2/python3 shebangs - unfortunately the most sensible solution that comes to my mind is: - move as much logic as possible away from shebanged scripts to libraries - rename the shebanged scripts to "script.py.in" - replace the python version in the shebang with m4 when building Changing python version as part of installation process is not a good idea IMO.
>> >> - configure.ac <http://configure.ac> must be adjusted: >> >> + AM_PATH_PYTHON([3.6]) >> + AC_PATH_PROG([PYTHON_CONFIG], [python3-config]) >> + AX_PYTHON_MODULE([six], [fatal], python3) >> >> It should be easy to make a patch for that.
I think that we should remove most of the python2/3 stuff from the configure.ac <http://configure.ac> such as PYTHON3_SUPPORT, fedora_run and all those stuff and decide which python version we are using in the spec file based on the OS and pass it as a parameter to the configure.
Again, I would like to state my opinion here - I think it would be nice if the configure parameter would only decide upon which RPMs we should build - py2 or py3. PYTHON3_SUPPORT is also used in makefiles to check if we can run py3 tests and I think it's a bad idea to take away the possibility to run both py2 and py3 tests from a single tree from developers. I would replace PYTHON3_SUPPORT with two vars - PYTHON2_SUPPORT and PYTHON3_SUPPORT - some distros are py2-only, some py2/py3, and some py3-only.
>> - The following changes in vdsm.spec.in <http://vdsm.spec.in> are needed: >> >> + Dependencies on Python packages must be adjusted. >> + python3_sitelib must be used instead of python_sitelib for Python 3. >> + Byte compiled files are placed into __pycache__ subdirectories by >> Python 3 and they must be installed. >>
we will do all that per package once we will start packaging
>> It would be nice to have patches for that so that we can build >> Python 3 rpm's easily. >> >> That's it at the moment. Most compatibility problems in Python code are >> easy to fix once they are identified. One important exception is bytes >> versus strings -- that's going to be delicate. >> >> The bytes versus strings problem is very present in communication >> code -- such as vdsm/rpc/http.py, vdsm/sslutils.py, etc. While strings >> constants are typically used in the code, the communication means work >> with bytes. This mismatch prevents Vdsm from communicating with the >> clients, so it would be very useful if someone could look at it and fix >> the code (one of the harder parts would probably be to decide where to >> keep bytes and where to already convert them to strings). >> >> HTH. As more people are involved in porting Vdsm to Python 3, it might >> be useful to coordinate all the efforts here, to avoid duplicate work, >> to share findings and to speed up the process. > > Indeed. We may also try to revive Trello usage. > Would you add relevant tasks to https://trello.com/b/tHeO1BFW/py3-vdsm ?
Sure, a couple of tasks added.
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/B4SMEAUJ34AQWA...
participants (4)
-
Dan Kenigsberg -
Gal Zaidman -
Marcin Sobczyk -
Milan Zamazal