
Hi all, Recent breakage in the vdsm CI flows have caused the change upload trigger to be disabled. This means that CI scores are no longer being automatically applied to uploaded changes. This means that patches cannot be merged into vdsm. I have a queue of patches which are otherwise ready for merge (which have passed CI in the past but needed rebasing). These patches have been stalled for almost a week now. What can we to to "unfreeze" the vdsm development process in the short and long term? Earlier today I worked with Sandro and David on manually running CI on my dev machine but am getting 100s of failures (so it looks like this wont even be a good short-term solution). -- Adam Litke

On Fri, Aug 21, 2015 at 05:04:03PM -0400, Adam Litke wrote:
Hi all,
Recent breakage in the vdsm CI flows have caused the change upload trigger to be disabled. This means that CI scores are no longer being automatically applied to uploaded changes. This means that patches cannot be merged into vdsm. I have a queue of patches which are otherwise ready for merge (which have passed CI in the past but needed rebasing). These patches have been stalled for almost a week now.
What can we to to "unfreeze" the vdsm development process in the short and long term? Earlier today I worked with Sandro and David on manually running CI on my dev machine but am getting 100s of failures (so it looks like this wont even be a good short-term solution).
Francesco and David were supposed to revive the broken job on Friday, but I see that it is still disabled. What is the reason of the delay?

--UHN/qo2QbUvPLonB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 08/23, Dan Kenigsberg wrote:
On Fri, Aug 21, 2015 at 05:04:03PM -0400, Adam Litke wrote:
Hi all, =20 Recent breakage in the vdsm CI flows have caused the change upload trigger to be disabled. This means that CI scores are no longer being automatically applied to uploaded changes. This means that patches cannot be merged into vdsm. I have a queue of patches which are otherwise ready for merge (which have passed CI in the past but needed rebasing). These patches have been stalled for almost a week now. =20 What can we to to "unfreeze" the vdsm development process in the short and long term? Earlier today I worked with Sandro and David on manually running CI on my dev machine but am getting 100s of failures (so it looks like this wont even be a good short-term solution). =20 Francesco and David were supposed to revive the broken job on Friday, but I see that it is still disabled. =20 What is the reason of the delay?
Francesco wanted the approval of the network guys, as the job has to create= the tun char device itself (because mock does not do it). If that's allowable, then it can be enabled right away (you'll have to reba= se to pull the scripts in). --=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --UHN/qo2QbUvPLonB Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJV2kLWAAoJEEBxx+HSYmnDdIEIAICNz+4nrJ4JALrLhkvPi32T n58Di54jJbKbyuTLx5vFVZrroRH8x0pHNo2xDS4kEP9v0qnFqGdmMCI/zLnUrfi2 dxC2x5WK8xUmRUW7OAkhlPyxY4EpfWXHwhRwebYYr8XNoTW0mwe5EQXE6ggUE3fW gvQr+wbJIoiMmZwM5PyMPHP6eqOEohTxFvqPo0LmJJDLWSWd9sddZ0/6ju/ilHgU aUMrEt4pUxEKXUoXyCP73CCOB18Kw3JQ6H8bKnNK9ptFyqMeQELIQec2Snj1ZBal EPVh19WUZ55GpTLBprKmYeNtDB99rZs5lZ3L45ca/H+lNHB8r3zAdnNTTgV0Zn4= =L8Nw -----END PGP SIGNATURE----- --UHN/qo2QbUvPLonB--

On Mon, Aug 24, 2015 at 12:01:58AM +0200, David Caro wrote:
On 08/23, Dan Kenigsberg wrote:
On Fri, Aug 21, 2015 at 05:04:03PM -0400, Adam Litke wrote:
Hi all,
Recent breakage in the vdsm CI flows have caused the change upload trigger to be disabled. This means that CI scores are no longer being automatically applied to uploaded changes. This means that patches cannot be merged into vdsm. I have a queue of patches which are otherwise ready for merge (which have passed CI in the past but needed rebasing). These patches have been stalled for almost a week now.
What can we to to "unfreeze" the vdsm development process in the short and long term? Earlier today I worked with Sandro and David on manually running CI on my dev machine but am getting 100s of failures (so it looks like this wont even be a good short-term solution).
Francesco and David were supposed to revive the broken job on Friday, but I see that it is still disabled.
What is the reason of the delay?
Francesco wanted the approval of the network guys, as the job has to create the tun char device itself (because mock does not do it).
If that's allowable, then it can be enabled right away (you'll have to rebase to pull the scripts in).
I'm not sure how this can be done. but in any case, I'd rather NOSE_EXCLUDE tcTests until we solve this issue.

----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "David Caro" <dcaro@redhat.com> Cc: "Adam Litke" <alitke@redhat.com>, fromani@redhat.com, devel@ovirt.org, "Sandro Bonazzola" <sbonazzo@redhat.com>, "David Caro Estevez" <dcaroest@redhat.com> Sent: Monday, August 24, 2015 8:53:07 AM Subject: Re: Merging patches into vdsm without CI
On Mon, Aug 24, 2015 at 12:01:58AM +0200, David Caro wrote:
On 08/23, Dan Kenigsberg wrote:
On Fri, Aug 21, 2015 at 05:04:03PM -0400, Adam Litke wrote:
Hi all,
Recent breakage in the vdsm CI flows have caused the change upload trigger to be disabled. This means that CI scores are no longer being automatically applied to uploaded changes. This means that patches cannot be merged into vdsm. I have a queue of patches which are otherwise ready for merge (which have passed CI in the past but needed rebasing). These patches have been stalled for almost a week now.
What can we to to "unfreeze" the vdsm development process in the short and long term? Earlier today I worked with Sandro and David on manually running CI on my dev machine but am getting 100s of failures (so it looks like this wont even be a good short-term solution).
Francesco and David were supposed to revive the broken job on Friday, but I see that it is still disabled.
What is the reason of the delay?
Francesco wanted the approval of the network guys, as the job has to create the tun char device itself (because mock does not do it).
If that's allowable, then it can be enabled right away (you'll have to rebase to pull the scripts in).
I'm not sure how this can be done. but in any case, I'd rather NOSE_EXCLUDE tcTests until we solve this issue.
Hi all, this is true. I didn't want to rush in a possible half-baked "solution" without any ack from the owners of the tests. With this patch: https://gerrit.ovirt.org/#/c/45134/3 the CI jobs WORKSFORME. I used these instructions http://www.ovirt.org/index.php?title=CI/Build_and_test_standards#Testing_the... and tested on el7 and fc22 succesfully. Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani

On Mon, Aug 24, 2015 at 03:32:53AM -0400, Francesco Romani wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "David Caro" <dcaro@redhat.com> Cc: "Adam Litke" <alitke@redhat.com>, fromani@redhat.com, devel@ovirt.org, "Sandro Bonazzola" <sbonazzo@redhat.com>, "David Caro Estevez" <dcaroest@redhat.com> Sent: Monday, August 24, 2015 8:53:07 AM Subject: Re: Merging patches into vdsm without CI
On Mon, Aug 24, 2015 at 12:01:58AM +0200, David Caro wrote:
On 08/23, Dan Kenigsberg wrote:
On Fri, Aug 21, 2015 at 05:04:03PM -0400, Adam Litke wrote:
Hi all,
Recent breakage in the vdsm CI flows have caused the change upload trigger to be disabled. This means that CI scores are no longer being automatically applied to uploaded changes. This means that patches cannot be merged into vdsm. I have a queue of patches which are otherwise ready for merge (which have passed CI in the past but needed rebasing). These patches have been stalled for almost a week now.
What can we to to "unfreeze" the vdsm development process in the short and long term? Earlier today I worked with Sandro and David on manually running CI on my dev machine but am getting 100s of failures (so it looks like this wont even be a good short-term solution).
Francesco and David were supposed to revive the broken job on Friday, but I see that it is still disabled.
What is the reason of the delay?
Francesco wanted the approval of the network guys, as the job has to create the tun char device itself (because mock does not do it).
If that's allowable, then it can be enabled right away (you'll have to rebase to pull the scripts in).
I'm not sure how this can be done. but in any case, I'd rather NOSE_EXCLUDE tcTests until we solve this issue.
Hi all,
this is true. I didn't want to rush in a possible half-baked "solution" without any ack from the owners of the tests.
With this patch: https://gerrit.ovirt.org/#/c/45134/3 the CI jobs WORKSFORME.
I used these instructions
http://www.ovirt.org/index.php?title=CI/Build_and_test_standards#Testing_the...
and tested on el7 and fc22 succesfully.
I've merged these patches (despite little comments that can wait for a follow-up patch). David, can you now enable CI on the master branch? Later, Francesco, would you backport these two patches to 3.6? Dan.

----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Francesco Romani" <fromani@redhat.com> Cc: "David Caro" <dcaro@redhat.com>, "Adam Litke" <alitke@redhat.com>, devel@ovirt.org, "Sandro Bonazzola" <sbonazzo@redhat.com>, "David Caro Estevez" <dcaroest@redhat.com> Sent: Monday, August 24, 2015 2:36:55 PM Subject: Re: Merging patches into vdsm without CI
On Mon, Aug 24, 2015 at 03:32:53AM -0400, Francesco Romani wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "David Caro" <dcaro@redhat.com> Cc: "Adam Litke" <alitke@redhat.com>, fromani@redhat.com, devel@ovirt.org, "Sandro Bonazzola" <sbonazzo@redhat.com>, "David Caro Estevez" <dcaroest@redhat.com> Sent: Monday, August 24, 2015 8:53:07 AM Subject: Re: Merging patches into vdsm without CI
On Mon, Aug 24, 2015 at 12:01:58AM +0200, David Caro wrote:
On 08/23, Dan Kenigsberg wrote:
On Fri, Aug 21, 2015 at 05:04:03PM -0400, Adam Litke wrote:
Hi all,
Recent breakage in the vdsm CI flows have caused the change upload trigger to be disabled. This means that CI scores are no longer being automatically applied to uploaded changes. This means that patches cannot be merged into vdsm. I have a queue of patches which are otherwise ready for merge (which have passed CI in the past but needed rebasing). These patches have been stalled for almost a week now.
What can we to to "unfreeze" the vdsm development process in the short and long term? Earlier today I worked with Sandro and David on manually running CI on my dev machine but am getting 100s of failures (so it looks like this wont even be a good short-term solution).
Francesco and David were supposed to revive the broken job on Friday, but I see that it is still disabled.
What is the reason of the delay?
Francesco wanted the approval of the network guys, as the job has to create the tun char device itself (because mock does not do it).
If that's allowable, then it can be enabled right away (you'll have to rebase to pull the scripts in).
I'm not sure how this can be done. but in any case, I'd rather NOSE_EXCLUDE tcTests until we solve this issue.
Hi all,
this is true. I didn't want to rush in a possible half-baked "solution" without any ack from the owners of the tests.
With this patch: https://gerrit.ovirt.org/#/c/45134/3 the CI jobs WORKSFORME.
I used these instructions
http://www.ovirt.org/index.php?title=CI/Build_and_test_standards#Testing_the...
and tested on el7 and fc22 succesfully.
I've merged these patches (despite little comments that can wait for a follow-up patch).
David, can you now enable CI on the master branch? Later, Francesco, would you backport these two patches to 3.6?
Sure thing! I'll just wait for Jenkins second check before to backport patches. Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani

----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Francesco Romani" <fromani@redhat.com> Cc: "David Caro" <dcaro@redhat.com>, "Adam Litke" <alitke@redhat.com>, = devel@ovirt.org, "Sandro Bonazzola" <sbonazzo@redhat.com>, "David Caro Estevez" <dcaroest@redhat.com> Sent: Monday, August 24, 2015 2:36:55 PM Subject: Re: Merging patches into vdsm without CI =20 On Mon, Aug 24, 2015 at 03:32:53AM -0400, Francesco Romani wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "David Caro" <dcaro@redhat.com> Cc: "Adam Litke" <alitke@redhat.com>, fromani@redhat.com, devel@ovirt.org, "Sandro Bonazzola" <sbonazzo@redhat.com>, "David Caro Estevez" <dcaroest@redhat.com> Sent: Monday, August 24, 2015 8:53:07 AM Subject: Re: Merging patches into vdsm without CI =20 On Mon, Aug 24, 2015 at 12:01:58AM +0200, David Caro wrote:
On 08/23, Dan Kenigsberg wrote:
On Fri, Aug 21, 2015 at 05:04:03PM -0400, Adam Litke wrote: > Hi all, >=20 > Recent breakage in the vdsm CI flows have caused the change u=
> trigger to be disabled. This means that CI scores are no lon= ger > being > automatically applied to uploaded changes. This means that p= atches > cannot be merged into vdsm. I have a queue of patches which = are > otherwise ready for merge (which have passed CI in the past b= ut > needed > rebasing). These patches have been stalled for almost a week= now. >=20 > What can we to to "unfreeze" the vdsm development process in =
--UugvWAfsgieZRqgk Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable The patch for the jenkins job is here: https://gerrit.ovirt.org/#/c/42477/ let me know if that's ok and I'll merge it (you might want for example run = it on more distros) On 08/24, Francesco Romani wrote: pload the
> short > and long term? Earlier today I worked with Sandro and David = on > manually running CI on my dev machine but am getting 100s of > failures > (so it looks like this wont even be a good short-term solutio= n). =20 Francesco and David were supposed to revive the broken job on F= riday, but I see that it is still disabled. =20 What is the reason of the delay? =20 Francesco wanted the approval of the network guys, as the job has= to create the tun char device itself (because mock does not do it). =20 =20 If that's allowable, then it can be enabled right away (you'll ha= ve to rebase to pull the scripts in). =20 I'm not sure how this can be done. but in any case, I'd rather NOSE_EXCLUDE tcTests until we solve this issue. =20 Hi all, =20 this is true. I didn't want to rush in a possible half-baked "solutio= n" without any ack from the owners of the tests. =20 With this patch: https://gerrit.ovirt.org/#/c/45134/3 the CI jobs WORKSFORME. =20 I used these instructions =20 http://www.ovirt.org/index.php?title=3DCI/Build_and_test_standards#Te= sting_the_scripts_locally =20 and tested on el7 and fc22 succesfully. =20 I've merged these patches (despite little comments that can wait for a follow-up patch). =20 David, can you now enable CI on the master branch? Later, Francesco, would you backport these two patches to 3.6? =20 Sure thing! I'll just wait for Jenkins second check before to backport pa= tches. =20 Bests, =20 --=20 Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --UugvWAfsgieZRqgk Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJV2x82AAoJEEBxx+HSYmnDr/kIAJR4uCySprd2d5LWcDrKmsnd hFJ6gbbMO0eo6Y23MNU6qF24B6i+CuVBRAdiN5nnZe7Ny/n+mgyjGea1DpSYuMyV IX7Gb0ADDXz/sUr4KrulyH6F3PQDo5sMRp4FimdC+Z5mOhdgUbtNzOvqFii2Fuxg 5N1WtMphH1qNuPKMBtTYE9JcE5fdsuCRP6/IclQEeAJN9zOze6C9TA+SzW3R2/oN SfN71cQ/i2DhhLL6L0HDS+pfJmUG3HX+A8lSAa/cTagwnqQOuyfMbVXxOzDWygc5 XJDrtbcXfeDOXmMYAgckMYuzcDseMUluHdqpVMWaj+M8vvkFzdAPRflAc/2HqLk= =kcFr -----END PGP SIGNATURE----- --UugvWAfsgieZRqgk--

----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Francesco Romani" <fromani@redhat.com> Cc: "David Caro" <dcaro@redhat.com>, "Adam Litke" <alitke@redhat.com>, = devel@ovirt.org, "Sandro Bonazzola" <sbonazzo@redhat.com>, "David Caro Estevez" <dcaroest@redhat.com> Sent: Monday, August 24, 2015 2:36:55 PM Subject: Re: Merging patches into vdsm without CI =20 On Mon, Aug 24, 2015 at 03:32:53AM -0400, Francesco Romani wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "David Caro" <dcaro@redhat.com> Cc: "Adam Litke" <alitke@redhat.com>, fromani@redhat.com, devel@ovirt.org, "Sandro Bonazzola" <sbonazzo@redhat.com>, "David Caro Estevez" <dcaroest@redhat.com> Sent: Monday, August 24, 2015 8:53:07 AM Subject: Re: Merging patches into vdsm without CI =20 On Mon, Aug 24, 2015 at 12:01:58AM +0200, David Caro wrote:
On 08/23, Dan Kenigsberg wrote:
On Fri, Aug 21, 2015 at 05:04:03PM -0400, Adam Litke wrote: > Hi all, >=20 > Recent breakage in the vdsm CI flows have caused the change u=
> trigger to be disabled. This means that CI scores are no lon= ger > being > automatically applied to uploaded changes. This means that p= atches > cannot be merged into vdsm. I have a queue of patches which = are > otherwise ready for merge (which have passed CI in the past b= ut > needed > rebasing). These patches have been stalled for almost a week= now. >=20 > What can we to to "unfreeze" the vdsm development process in =
--pvezYHf7grwyp3Bc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I've enabled the job that the jenkins patch creates, created manually, but review the patch so I can merge it please: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc22-x86_64/ On 08/24, Francesco Romani wrote: pload the
> short > and long term? Earlier today I worked with Sandro and David = on > manually running CI on my dev machine but am getting 100s of > failures > (so it looks like this wont even be a good short-term solutio= n). =20 Francesco and David were supposed to revive the broken job on F= riday, but I see that it is still disabled. =20 What is the reason of the delay? =20 Francesco wanted the approval of the network guys, as the job has= to create the tun char device itself (because mock does not do it). =20 =20 If that's allowable, then it can be enabled right away (you'll ha= ve to rebase to pull the scripts in). =20 I'm not sure how this can be done. but in any case, I'd rather NOSE_EXCLUDE tcTests until we solve this issue. =20 Hi all, =20 this is true. I didn't want to rush in a possible half-baked "solutio= n" without any ack from the owners of the tests. =20 With this patch: https://gerrit.ovirt.org/#/c/45134/3 the CI jobs WORKSFORME. =20 I used these instructions =20 http://www.ovirt.org/index.php?title=3DCI/Build_and_test_standards#Te= sting_the_scripts_locally =20 and tested on el7 and fc22 succesfully. =20 I've merged these patches (despite little comments that can wait for a follow-up patch). =20 David, can you now enable CI on the master branch? Later, Francesco, would you backport these two patches to 3.6? =20 Sure thing! I'll just wait for Jenkins second check before to backport pa= tches. =20 Bests, =20 --=20 Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --pvezYHf7grwyp3Bc Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJV2x99AAoJEEBxx+HSYmnDhrIH/0OBBMRUMuMSGNt9G47VEMUK 5sMpIyq+VE4Cb+whQAB0MD2P6WSqxAVDvmlAbEmpLzFKdj6mx5hlubasmAajWgCa +RrgOpl44LpYzkLALa48rJReNpzZxm1qSsQvBL0yxSJ/XjqyalJF2HwTVBSnG0i4 bAW7wgKE2IHrMUcTEwJAfUe2Lca2WJT1lI68ytus57HNgp7G0zWhFBXH2A2raVoz HDxYz1waVmC3l3QfyaYorootYws+3j18RsB7TGkO34BIgY3vHRamb+wpwXnLU1E4 XzwA5Tl1EFHrKuIMivx3PzrkEk+sJQG4XiERcmhvjFGpccwN3Ekqmnjt5JVyFRE= =5CUo -----END PGP SIGNATURE----- --pvezYHf7grwyp3Bc--

On Mon, Aug 24, 2015 at 03:43:25PM +0200, David Caro wrote:
I've enabled the job that the jenkins patch creates, created manually, but review the patch so I can merge it please:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc22-x86_64/
Thanks. We've now taken automation/check-patch.sh into ovirt-3.6. Can you issue a similar job for 3.6? It would be nice to run in on el7. Regards, Dan

--DrWhICOqskFTAXiy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Sent patch and verifyed: https://gerrit.ovirt.org/#/c/45281 On 08/25, Dan Kenigsberg wrote:
On Mon, Aug 24, 2015 at 03:43:25PM +0200, David Caro wrote:
=20 I've enabled the job that the jenkins patch creates, created manually, = but review the patch so I can merge it please: =20 http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc22-x86_64/ =20 Thanks. =20 We've now taken automation/check-patch.sh into ovirt-3.6. Can you issue a similar job for 3.6? It would be nice to run in on el7. =20 Regards, Dan
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --DrWhICOqskFTAXiy Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJV3CO1AAoJEEBxx+HSYmnD57sH/0r60gBxRQYLsHvHOhEVWwdB CWp035TDGERjMkRkS8ARGajrR/1E+Diqblqpl76hDmbURWfB/pGiidKBTl3f0rlF Vc2KMpbg/yZOoElxaJD+kKV0W+2X9W/rmLWygdHDM0EjlLcu+CXdq/ZiIh30LdG3 5CMk2s3VWIl5DMDZa8V1u1wUN+veH/OCuF744PG8+uN8Lm3B1Y1CmhMdYHA6L+wx T7CQPotbNQpnlGM2qqjxnMOPDvxQ2WpxLgLfQgKw4EyT0idiFh/Wka86jHQlCD02 Cn+KCrjg5KHiUwx17IG2c390CGn9/0N34L24FatYOWnlN7eKKpL9WLD0IsEE/9s= =lc46 -----END PGP SIGNATURE----- --DrWhICOqskFTAXiy--
participants (4)
-
Adam Litke
-
Dan Kenigsberg
-
David Caro
-
Francesco Romani