On Mon, Feb 10, 2020 at 4:13 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/346/ lago.log has: 1. some repo issue: 2020-02-10 02:11:13,681::ERROR::repoman.common.parser::No artifacts found for source /var/lib/lago/ovirt-appliance-4.3-el7:only-missing Galit - any idea? 2. IPv6 issue - after a long series of tracebacks: libvirtError: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable Dominik, any idea? Not sure if this is an infra issue or a recent change to OST (or lago, or libvirt...). Thanks and best regards, -- Didi _______________________________________________ Infra mailing list -- infra(a)ovirt.org To unsubscribe send an email to infra-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/URERUP2VUCW...

Checking this Once I have more info I will update. On Mon, Feb 10, 2020 at 9:16 AM Parth Dhanjal <dparth(a)redhat.com&gt; wrote: > Hey! > > hc_basic_suite_4.3 is failing with the same error as well > > Project: > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > Build: > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > <https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > l > > *07:36:24* @ Start Prefix: *07:36:24* # Start nets: *07:36:24* * Create network lago-hc-basic-suite-4-3-net-management: *07:36:30* * Create network lago-hc-basic-suite-4-3-net-management: ERROR (in 0:00:05)*07:36:30* # Start nets: ERROR (in 0:00:05)*07:36:30* @ Start Prefix: ERROR (in 0:00:05)*07:36:30* Error occured, aborting*07:36:30* Traceback (most recent call last):*07:36:30* File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 969, in main*07:36:30* cli_plugins[args.verb].do_run(args)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run*07:36:30* self._do_run(**vars(args))*07:36:30* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 573, in wrapper*07:36:30* return func(*args, **kwargs)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 584, in wrapper*07:36:30* return func(*args, prefix=prefix, **kwargs)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 271, in do_start*07:36:30* prefix.start(vm_names=vm_names)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/sdk_utils.py", line 50, in wrapped*07:36:30* return func(*args, **kwargs)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1323, in start*07:36:30* self.virt_env.start(vm_names=vm_names)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/virt.py", line 341, in start*07:36:30* net.start()*07:36:30* File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/network.py", line 115, in start*07:36:30* net = self.libvirt_con.networkCreateXML(self._libvirt_xml())*07:36:30* File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4216, in networkCreateXML*07:36:30* if ret is None:raise libvirtError('virNetworkCreateXML() failed', conn=self)*07:36:30* libvirtError: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable*07:36:30* + on_exit*07:36:30* + [[ 1 -ne 0 ]]*07:36:30* + logger.error 'on_exit: Exiting with a non-zero status'*07:36:30* + logger.log ERROR 'on_exit: Exiting with a non-zero status'*07:36:30* + set +x*07:36:30* 2020-02-10 02:06:30.296966051+0000 run_suite.sh::on_exit::ERROR:: on_exit: Exiting with a non-zero status > > > > On Mon, Feb 10, 2020 at 12:08 PM Yedidyah Bar David <didi(a)redhat.com&gt; > wrote: > >> On Mon, Feb 10, 2020 at 4:13 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: >> > >> > Project: >> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/ >> > Build: >> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/346/ >> >> lago.log has: >> >> 1. some repo issue: >> >> 2020-02-10 02:11:13,681::ERROR::repoman.common.parser::No artifacts >> found for source /var/lib/lago/ovirt-appliance-4.3-el7:only-missing >> >> Galit - any idea? >> >> 2. IPv6 issue - after a long series of tracebacks: >> >> libvirtError: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid >> backend or is unavailable >> >> Dominik, any idea? Not sure if this is an infra issue or a recent >> change to OST (or lago, or libvirt...). >> >> Thanks and best regards, >> -- >> Didi >> _______________________________________________ >> Infra mailing list -- infra(a)ovirt.org >> To unsubscribe send an email to infra-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/URERUP2VUCW... >> > -- GALIT ROSENTHAL SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit(a)redhat.com T: 972-9-7692230 <https://red.ht/sig>

On Mon, Feb 10, 2020 at 10:32 AM Dominik Holler <dholler(a)redhat.com&gt; wrote: > Hello, > is this issue reproducible local for someone, or does this happen only on > jenkins? > > For me it seems to happen only on jenkins. I don't know enough about the environment on jenkins. Might a "yum update" fix the problem?

I guess that there are incompatible versions of libvirt and firewalld installed.

> On Mon, Feb 10, 2020 at 9:46 AM Galit Rosenthal <grosenth(a)redhat.com&gt; > wrote: > >> Checking this >> Once I have more info I will update. >> >> On Mon, Feb 10, 2020 at 9:16 AM Parth Dhanjal <dparth(a)redhat.com&gt; wrote: >> >>> Hey! >>> >>> hc_basic_suite_4.3 is failing with the same error as well >>> >>> Project: >>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>> Build: >>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>> <https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>> l >>> >>> *07:36:24* @ Start Prefix: *07:36:24* # Start nets: *07:36:24* * Create network lago-hc-basic-suite-4-3-net-management: *07:36:30* * Create network lago-hc-basic-suite-4-3-net-management: ERROR (in 0:00:05)*07:36:30* # Start nets: ERROR (in 0:00:05)*07:36:30* @ Start Prefix: ERROR (in 0:00:05)*07:36:30* Error occured, aborting*07:36:30* Traceback (most recent call last):*07:36:30* File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 969, in main*07:36:30* cli_plugins[args.verb].do_run(args)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run*07:36:30* self._do_run(**vars(args))*07:36:30* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 573, in wrapper*07:36:30* return func(*args, **kwargs)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 584, in wrapper*07:36:30* return func(*args, prefix=prefix, **kwargs)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 271, in do_start*07:36:30* prefix.start(vm_names=vm_names)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/sdk_utils.py", line 50, in wrapped*07:36:30* return func(*args, **kwargs)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1323, in start*07:36:30* self.virt_env.start(vm_names=vm_names)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/virt.py", line 341, in start*07:36:30* net.start()*07:36:30* File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/network.py", line 115, in start*07:36:30* net = self.libvirt_con.networkCreateXML(self._libvirt_xml())*07:36:30* File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4216, in networkCreateXML*07:36:30* if ret is None:raise libvirtError('virNetworkCreateXML() failed', conn=self)*07:36:30* libvirtError: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable*07:36:30* + on_exit*07:36:30* + [[ 1 -ne 0 ]]*07:36:30* + logger.error 'on_exit: Exiting with a non-zero status'*07:36:30* + logger.log ERROR 'on_exit: Exiting with a non-zero status'*07:36:30* + set +x*07:36:30* 2020-02-10 02:06:30.296966051+0000 run_suite.sh::on_exit::ERROR:: on_exit: Exiting with a non-zero status >>> >>> >>> >>> On Mon, Feb 10, 2020 at 12:08 PM Yedidyah Bar David <didi(a)redhat.com&gt; >>> wrote: >>> >>>> On Mon, Feb 10, 2020 at 4:13 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: >>>> > >>>> > Project: >>>> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/ >>>> > Build: >>>> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/346/ >>>> >>>> lago.log has: >>>> >>>> 1. some repo issue: >>>> >>>> 2020-02-10 02:11:13,681::ERROR::repoman.common.parser::No artifacts >>>> found for source /var/lib/lago/ovirt-appliance-4.3-el7:only-missing >>>> >>>> Galit - any idea? >>>> >>>> 2. IPv6 issue - after a long series of tracebacks: >>>> >>>> libvirtError: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid >>>> backend or is unavailable >>>> >>>> Dominik, any idea? Not sure if this is an infra issue or a recent >>>> change to OST (or lago, or libvirt...). >>>> >>>> Thanks and best regards, >>>> -- >>>> Didi >>>> _______________________________________________ >>>> Infra mailing list -- infra(a)ovirt.org >>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/URERUP2VUCW... >>>> >>> >> >> -- >> >> GALIT ROSENTHAL >> >> SOFTWARE ENGINEER >> >> Red Hat >> >> <https://www.redhat.com/> >> >> galit(a)redhat.com T: 972-9-7692230 >> <https://red.ht/sig> >> > _______________________________________________ Infra mailing list -- infra(a)ovirt.org To unsubscribe send an email to infra-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/SYYL7RVRKNW...

On Mon, Feb 10, 2020 at 3:19 PM Barak Korren <bkorren(a)redhat.com&gt; wrote: > > > On Mon, 10 Feb 2020 at 15:11, Dominik Holler <dholler(a)redhat.com&gt; wrote: > >> >> On Mon, Feb 10, 2020 at 10:32 AM Dominik Holler <dholler(a)redhat.com&gt; >> wrote: >> >>> Hello, >>> is this issue reproducible local for someone, or does this happen only >>> on jenkins? >>> >>> >> For me it seems to happen only on jenkins. >> I don't know enough about the environment on jenkins. >> Might a "yum update" fix the problem? >> > > > `yum update` actually caused the problem... > > >> I guess that there are incompatible versions of libvirt and firewalld >> installed. >> > > We have the latest versions for both, currently we suspect its the > startup order > > >> >> >>> On Mon, Feb 10, 2020 at 9:46 AM Galit Rosenthal <grosenth(a)redhat.com&gt; >>> wrote: >>> >>>> Checking this >>>> Once I have more info I will update. >>>> >>>> On Mon, Feb 10, 2020 at 9:16 AM Parth Dhanjal <dparth(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Hey! >>>>> >>>>> hc_basic_suite_4.3 is failing with the same error as well >>>>> >>>>> Project: >>>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>>>> Build: >>>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>>>> <https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>>>> l >>>>> >>>>> *07:36:24* @ Start Prefix: *07:36:24* # Start nets: *07:36:24* * Create network lago-hc-basic-suite-4-3-net-management: *07:36:30* * Create network lago-hc-basic-suite-4-3-net-management: ERROR (in 0:00:05)*07:36:30* # Start nets: ERROR (in 0:00:05)*07:36:30* @ Start Prefix: ERROR (in 0:00:05)*07:36:30* Error occured, aborting*07:36:30* Traceback (most recent call last):*07:36:30* File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 969, in main*07:36:30* cli_plugins[args.verb].do_run(args)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run*07:36:30* self._do_run(**vars(args))*07:36:30* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 573, in wrapper*07:36:30* return func(*args, **kwargs)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 584, in wrapper*07:36:30* return func(*args, prefix=prefix, **kwargs)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 271, in do_start*07:36:30* prefix.start(vm_names=vm_names)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/sdk_utils.py", line 50, in wrapped*07:36:30* return func(*args, **kwargs)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1323, in start*07:36:30* self.virt_env.start(vm_names=vm_names)*07:36:30* File "/usr/lib/python2.7/site-packages/lago/virt.py", line 341, in start*07:36:30* net.start()*07:36:30* File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/network.py", line 115, in start*07:36:30* net = self.libvirt_con.networkCreateXML(self._libvirt_xml())*07:36:30* File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4216, in networkCreateXML*07:36:30* if ret is None:raise libvirtError('virNetworkCreateXML() failed', conn=self)*07:36:30* libvirtError: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid backend or is unavailable >>>>> >>>>> I now spent some time and think I at least managed to find what emits this error: https://github.com/firewalld/firewalld/blob/master/src/firewall/core/fw.py So perhaps it's an update to firewalld, or to iptables, ip6tables, etc.

> *07:36:30* + on_exit*07:36:30* + [[ 1 -ne 0 ]]*07:36:30* + logger.error 'on_exit: Exiting with a non-zero status'*07:36:30* + logger.log ERROR 'on_exit: Exiting with a non-zero status'*07:36:30* + set +x*07:36:30* 2020-02-10 02:06:30.296966051+0000 run_suite.sh::on_exit::ERROR:: on_exit: Exiting with a non-zero status >>>>> >>>>> >>>>> >>>>> On Mon, Feb 10, 2020 at 12:08 PM Yedidyah Bar David <didi(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> On Mon, Feb 10, 2020 at 4:13 AM <jenkins(a)jenkins.phx.ovirt.org&gt; >>>>>> wrote: >>>>>> > >>>>>> > Project: >>>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/ >>>>>> > Build: >>>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/346/ >>>>>> >>>>>> lago.log has: >>>>>> >>>>>> 1. some repo issue: >>>>>> >>>>>> 2020-02-10 02:11:13,681::ERROR::repoman.common.parser::No artifacts >>>>>> found for source /var/lib/lago/ovirt-appliance-4.3-el7:only-missing >>>>>> >>>>>> Galit - any idea? >>>>>> >>>>>> 2. IPv6 issue - after a long series of tracebacks: >>>>>> >>>>>> libvirtError: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid >>>>>> backend or is unavailable >>>>>> >>>>>> Dominik, any idea? Not sure if this is an infra issue or a recent >>>>>> change to OST (or lago, or libvirt...). >>>>>> >>>>>> Thanks and best regards, >>>>>> -- >>>>>> Didi >>>>>> _______________________________________________ >>>>>> Infra mailing list -- infra(a)ovirt.org >>>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/URERUP2VUCW... >>>>>> >>>>> >>>> >>>> -- >>>> >>>> GALIT ROSENTHAL >>>> >>>> SOFTWARE ENGINEER >>>> >>>> Red Hat >>>> >>>> <https://www.redhat.com/> >>>> >>>> galit(a)redhat.com T: 972-9-7692230 >>>> <https://red.ht/sig> >>>> >>> _______________________________________________ >> Infra mailing list -- infra(a)ovirt.org >> To unsubscribe send an email to infra-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/SYYL7RVRKNW... >> > > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/574TXW4JBJG... > -- Didi