VDSM build failure: unsupported pickle protocol: 5
hi, I'm trying to enable copr builds for vdsm ( https://gerrit.ovirt.org/c/vdsm/+/117368 ) And it's currently failing to rebuild src.rpm (generated on Fedora 34) for el8 with the following error: ( https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapsh... ) make[2]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' Making all in api make[3]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' Generate vdsm-api.html chmod u+w . PYTHONPATH=./../../:./../../vdsm \ ./schema_to_html.py vdsm-api ./vdsm-api.html Traceback (most recent call last): File "./schema_to_html.py", line 250, in <module> main() File "./schema_to_html.py", line 245, in main api_schema = vdsmapi.Schema((schema_type,), strict_mode=False) File "/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api/vdsmapi.py", line 145, in __init__ loaded_schema = pickle.loads(f.read()) ValueError: unsupported pickle protocol: 5 make[3]: *** [Makefile:697: vdsm-api.html] Error 1 make[3]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' make[2]: *** [Makefile:644: all-recursive] Error 1 make[2]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' make[1]: *** [Makefile:466: all-recursive] Error 1 make[1]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib' make: *** [Makefile:539: all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.nDfLzv (%build) Sounds like the make dist process ran on Fedora 34 brings a source file used at build time on el8 in a cpickle format which is not backward compatible. Seems to be a bug, the cpickle shouldn't be included in the tar.gz, it should be generated at build time. Comments? -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*
Hi, On 10/28/21 11:59, Sandro Bonazzola wrote:
hi, I'm trying to enable copr builds for vdsm ( https://gerrit.ovirt.org/c/vdsm/+/117368 <https://gerrit.ovirt.org/c/vdsm/+/117368> )
And it's currently failing to rebuild src.rpm (generated on Fedora 34) for el8 with the following error: (https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapsh... <https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-8-x86_64/02912480-vdsm/build.log.gz> )
make[2]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' Making all in api make[3]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' Generate vdsm-api.html chmod u+w . PYTHONPATH=./../../:./../../vdsm \ ./schema_to_html.py vdsm-api ./vdsm-api.html Traceback (most recent call last): File "./schema_to_html.py", line 250, in <module> main() File "./schema_to_html.py", line 245, in main api_schema = vdsmapi.Schema((schema_type,), strict_mode=False) File "/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api/vdsmapi.py", line 145, in __init__ loaded_schema = pickle.loads(f.read()) ValueError: unsupported pickle protocol: 5 make[3]: *** [Makefile:697: vdsm-api.html] Error 1 make[3]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' make[2]: *** [Makefile:644: all-recursive] Error 1 make[2]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' make[1]: *** [Makefile:466: all-recursive] Error 1 make[1]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib' make: *** [Makefile:539: all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.nDfLzv (%build)
Sounds like the make dist process ran on Fedora 34 brings a source file used at build time on el8 in a cpickle format which is not backward compatible.
Seems to be a bug, the cpickle shouldn't be included in the tar.gz, it should be generated at build time.
Comments?
The pickles were introduced here: https://gerrit.ovirt.org/c/vdsm/+/94196 AFAIR they were added to the vdsm-api package because previously we were generating them in during rpm installation in %post section which caused issues with oVirt Node. I'm not sure how easy will it be to not have them in the srpm/tar.gz but have them in the rpm. As a quick workaround I can change the pickle protocol version that we use [1] to '4', which should work for both Fedora 34 and el8. Regards, Marcin [1] https://github.com/oVirt/vdsm/blob/1969ab99c371ad498ea8693671cec60e2d0d49c2/...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <mailto:sbonazzo@redhat.com>
** *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. * *
*
Hi, I have posted a patch with the suggested workaround [1]. If need anything else, please let me know. [1] https://gerrit.ovirt.org/c/vdsm/+/117372 Harel On Thu, Oct 28, 2021 at 2:08 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
On 10/28/21 11:59, Sandro Bonazzola wrote:
hi, I'm trying to enable copr builds for vdsm ( https://gerrit.ovirt.org/c/vdsm/+/117368 <https://gerrit.ovirt.org/c/vdsm/+/117368> )
And it's currently failing to rebuild src.rpm (generated on Fedora 34) for el8 with the following error: ( https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapsh... < https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-8-x86_64/02912480-vdsm/build.log.gz>
)
make[2]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' Making all in api make[3]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' Generate vdsm-api.html chmod u+w . PYTHONPATH=./../../:./../../vdsm \ ./schema_to_html.py vdsm-api ./vdsm-api.html Traceback (most recent call last): File "./schema_to_html.py", line 250, in <module> main() File "./schema_to_html.py", line 245, in main api_schema = vdsmapi.Schema((schema_type,), strict_mode=False) File "/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api/vdsmapi.py", line 145, in __init__ loaded_schema = pickle.loads(f.read()) ValueError: unsupported pickle protocol: 5 make[3]: *** [Makefile:697: vdsm-api.html] Error 1 make[3]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' make[2]: *** [Makefile:644: all-recursive] Error 1 make[2]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' make[1]: *** [Makefile:466: all-recursive] Error 1 make[1]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib' make: *** [Makefile:539: all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.nDfLzv (%build)
Sounds like the make dist process ran on Fedora 34 brings a source file used at build time on el8 in a cpickle format which is not backward compatible.
Seems to be a bug, the cpickle shouldn't be included in the tar.gz, it should be generated at build time.
Comments?
The pickles were introduced here:
https://gerrit.ovirt.org/c/vdsm/+/94196
AFAIR they were added to the vdsm-api package because previously we were generating them in during rpm installation in %post section which caused issues with oVirt Node.
I'm not sure how easy will it be to not have them in the srpm/tar.gz but have them in the rpm. As a quick workaround I can change the pickle protocol version that we use [1] to '4', which should work for both Fedora 34 and el8.
Regards, Marcin
[1]
https://github.com/oVirt/vdsm/blob/1969ab99c371ad498ea8693671cec60e2d0d49c2/...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <mailto:sbonazzo@redhat.com>
** *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. * *
*
verified, this works for me Il giorno gio 28 ott 2021 alle ore 13:41 Harel Braha <hbraha@redhat.com> ha scritto:
Hi, I have posted a patch with the suggested workaround [1]. If need anything else, please let me know.
[1] https://gerrit.ovirt.org/c/vdsm/+/117372
Harel
On Thu, Oct 28, 2021 at 2:08 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
On 10/28/21 11:59, Sandro Bonazzola wrote:
hi, I'm trying to enable copr builds for vdsm ( https://gerrit.ovirt.org/c/vdsm/+/117368 <https://gerrit.ovirt.org/c/vdsm/+/117368> )
And it's currently failing to rebuild src.rpm (generated on Fedora 34) for el8 with the following error: ( https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapsh... < https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-8-x86_64/02912480-vdsm/build.log.gz>
)
make[2]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' Making all in api make[3]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' Generate vdsm-api.html chmod u+w . PYTHONPATH=./../../:./../../vdsm \ ./schema_to_html.py vdsm-api ./vdsm-api.html Traceback (most recent call last): File "./schema_to_html.py", line 250, in <module> main() File "./schema_to_html.py", line 245, in main api_schema = vdsmapi.Schema((schema_type,), strict_mode=False) File "/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api/vdsmapi.py", line 145, in __init__ loaded_schema = pickle.loads(f.read()) ValueError: unsupported pickle protocol: 5 make[3]: *** [Makefile:697: vdsm-api.html] Error 1 make[3]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' make[2]: *** [Makefile:644: all-recursive] Error 1 make[2]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' make[1]: *** [Makefile:466: all-recursive] Error 1 make[1]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib' make: *** [Makefile:539: all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.nDfLzv (%build)
Sounds like the make dist process ran on Fedora 34 brings a source file used at build time on el8 in a cpickle format which is not backward compatible.
Seems to be a bug, the cpickle shouldn't be included in the tar.gz, it should be generated at build time.
Comments?
The pickles were introduced here:
https://gerrit.ovirt.org/c/vdsm/+/94196
AFAIR they were added to the vdsm-api package because previously we were generating them in during rpm installation in %post section which caused issues with oVirt Node.
I'm not sure how easy will it be to not have them in the srpm/tar.gz but have them in the rpm. As a quick workaround I can change the pickle protocol version that we use [1] to '4', which should work for both Fedora 34 and el8.
Regards, Marcin
[1]
https://github.com/oVirt/vdsm/blob/1969ab99c371ad498ea8693671cec60e2d0d49c2/...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <mailto:sbonazzo@redhat.com>
** *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. * *
*
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*
On Thu, Oct 28, 2021 at 2:08 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
On 10/28/21 11:59, Sandro Bonazzola wrote:
hi, I'm trying to enable copr builds for vdsm ( https://gerrit.ovirt.org/c/vdsm/+/117368 <https://gerrit.ovirt.org/c/vdsm/+/117368> )
And it's currently failing to rebuild src.rpm (generated on Fedora 34) for el8 with the following error: (https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapsh... <https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-8-x86_64/02912480-vdsm/build.log.gz> )
make[2]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' Making all in api make[3]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' Generate vdsm-api.html chmod u+w . PYTHONPATH=./../../:./../../vdsm \ ./schema_to_html.py vdsm-api ./vdsm-api.html Traceback (most recent call last): File "./schema_to_html.py", line 250, in <module> main() File "./schema_to_html.py", line 245, in main api_schema = vdsmapi.Schema((schema_type,), strict_mode=False) File "/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api/vdsmapi.py", line 145, in __init__ loaded_schema = pickle.loads(f.read()) ValueError: unsupported pickle protocol: 5 make[3]: *** [Makefile:697: vdsm-api.html] Error 1 make[3]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' make[2]: *** [Makefile:644: all-recursive] Error 1 make[2]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' make[1]: *** [Makefile:466: all-recursive] Error 1 make[1]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib' make: *** [Makefile:539: all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.nDfLzv (%build)
Sounds like the make dist process ran on Fedora 34 brings a source file used at build time on el8 in a cpickle format which is not backward compatible.
Seems to be a bug, the cpickle shouldn't be included in the tar.gz, it should be generated at build time.
Comments?
The pickles were introduced here:
https://gerrit.ovirt.org/c/vdsm/+/94196
AFAIR they were added to the vdsm-api package because previously we were generating them in during rpm installation in %post section which caused issues with oVirt Node.
I'm not sure how easy will it be to not have them in the srpm/tar.gz but have them in the rpm. As a quick workaround I can change the pickle protocol version that we use [1] to '4', which should work for both Fedora 34 and el8.
Creating the pickle when building srpm is bad. This means you must use some old (slow) pickle protocol that works on all possible platforms, instead of the highest (fast) protocol available on the target platform. We should really create the pickle when building the rpm, which is done in mock, with the right python version. If we cannot do this then building the cache on the host during configure time will be an easy solution. Check if the cache exists in /var/lib/cache/vdsm/... and regenerate it if needed. If we don't want to run the schema_to_html tool at this time, we can add the schema in json format - this can be done when building the srpm. Converting json to pickle is very fast and does not have any dependencies. Nir
Regards, Marcin
[1] https://github.com/oVirt/vdsm/blob/1969ab99c371ad498ea8693671cec60e2d0d49c2/...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <mailto:sbonazzo@redhat.com>
** *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. * *
*
On 10/28/21 18:52, Nir Soffer wrote:
On Thu, Oct 28, 2021 at 2:08 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
On 10/28/21 11:59, Sandro Bonazzola wrote:
hi, I'm trying to enable copr builds for vdsm ( https://gerrit.ovirt.org/c/vdsm/+/117368 <https://gerrit.ovirt.org/c/vdsm/+/117368> )
And it's currently failing to rebuild src.rpm (generated on Fedora 34) for el8 with the following error: (https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapsh... <https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-8-x86_64/02912480-vdsm/build.log.gz> )
make[2]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' Making all in api make[3]: Entering directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' Generate vdsm-api.html chmod u+w . PYTHONPATH=./../../:./../../vdsm \ ./schema_to_html.py vdsm-api ./vdsm-api.html Traceback (most recent call last): File "./schema_to_html.py", line 250, in <module> main() File "./schema_to_html.py", line 245, in main api_schema = vdsmapi.Schema((schema_type,), strict_mode=False) File "/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api/vdsmapi.py", line 145, in __init__ loaded_schema = pickle.loads(f.read()) ValueError: unsupported pickle protocol: 5 make[3]: *** [Makefile:697: vdsm-api.html] Error 1 make[3]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm/api' make[2]: *** [Makefile:644: all-recursive] Error 1 make[2]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib/vdsm' make[1]: *** [Makefile:466: all-recursive] Error 1 make[1]: Leaving directory '/builddir/build/BUILD/vdsm-4.50.0.1/lib' make: *** [Makefile:539: all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.nDfLzv (%build)
Sounds like the make dist process ran on Fedora 34 brings a source file used at build time on el8 in a cpickle format which is not backward compatible.
Seems to be a bug, the cpickle shouldn't be included in the tar.gz, it should be generated at build time.
Comments?
The pickles were introduced here:
https://gerrit.ovirt.org/c/vdsm/+/94196
AFAIR they were added to the vdsm-api package because previously we were generating them in during rpm installation in %post section which caused issues with oVirt Node.
I'm not sure how easy will it be to not have them in the srpm/tar.gz but have them in the rpm. As a quick workaround I can change the pickle protocol version that we use [1] to '4', which should work for both Fedora 34 and el8.
Creating the pickle when building srpm is bad. This means you must use some old (slow) pickle protocol that works on all possible platforms, instead of the highest (fast) protocol available on the target platform.
We should really create the pickle when building the rpm, which is done in mock, with the right python version.
If we cannot do this then building the cache on the host during configure time will be an easy solution. Check if the cache exists in /var/lib/cache/vdsm/... and regenerate it if needed.
If we don't want to run the schema_to_html tool at this time, we can add the schema in json format - this can be done when building the srpm. Converting json to pickle is very fast and does not have any dependencies.
Ack, let's have this quick workaround for now and implement a proper solution later. Regards, Marcin
Nir
Regards, Marcin
[1] https://github.com/oVirt/vdsm/blob/1969ab99c371ad498ea8693671cec60e2d0d49c2/...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <mailto:sbonazzo@redhat.com>
** *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. * *
*
participants (4)
-
Harel Braha -
Marcin Sobczyk -
Nir Soffer -
Sandro Bonazzola