Update: artifactory problem is resolved (incident report)

[Eyal Edri]

Hi,

Today jenkins.ovirt.org had suffered a major incident which occured due to bad syncing from maven sonatype.
it all started from the jboss commit "core: CDI workaournd jboss 7.1.1 bug", which was fixed later by roy,
but for some reason (might be sonatype repo deleting ovirt artifacts), we were left with outdated jars in all jenkins local
repos and on artifactory.ovirt.org.

roy sent this morning how it should be resolved: 
 - build is failing because ovirt-host-deploy needs refresh using -U .
   alonbl updates its artifacts just now so it should take a while to refresh

but it didn't work.

also didn't work:
  - cleaning cache from artifactory
  - deleting local maven repos from jenkins jobs

what did work finally:
 - deleting the sonatype from artifactory.ovirt.org and creating it again, plug updating ovirt-mirror virtual as well.
 - some jobs might need to be updated/deleted local WS manually still - please report to infra on jobs that still fail.

we probably need to think how to handle such incidents in the future (if sonatype continues to delete our jars...).
any suggestions will be gladly considered.

oVirt infra team.


----- Original Message -----
> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> To: "Ohad Basan" <obasan at redhat.com>, "infra" <infra at ovirt.org>
> Sent: Tuesday, June 24, 2014 6:22:35 PM
> Subject: Re: artifactory problem is resolved
> 
> Il 24/06/2014 17:10, Ohad Basan ha scritto:
> > Hello
> > 
> > the artifactory problem we had today (fetching the wrong host deploy)
> > should be fixed
> > send an email if you hit more problems
> 
> http://jenkins.ovirt.org/job/ovirt-engine_master_create-rpms-quick_gerrit/2804/label=fedora20/console
> 
> > 
> > thanks,
> > 
> > Ohad
> > _______________________________________________
> > Infra mailing list
> > Infra at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> > 
> 
> 
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>