[ovirt-devel] Update: artifactory problem is resolved (incident report)

Hi, Today jenkins.ovirt.org had suffered a major incident which occured due to bad syncing from maven sonatype. it all started from the jboss commit "core: CDI workaournd jboss 7.1.1 bug", which was fixed later by roy, but for some reason (might be sonatype repo deleting ovirt artifacts), we were left with outdated jars in all jenkins local repos and on artifactory.ovirt.org. roy sent this morning how it should be resolved: - build is failing because ovirt-host-deploy needs refresh using -U . alonbl updates its artifacts just now so it should take a while to refresh but it didn't work. also didn't work: - cleaning cache from artifactory - deleting local maven repos from jenkins jobs what did work finally: - deleting the sonatype from artifactory.ovirt.org and creating it again, plug updating ovirt-mirror virtual as well. - some jobs might need to be updated/deleted local WS manually still - please report to infra on jobs that still fail. we probably need to think how to handle such incidents in the future (if sonatype continues to delete our jars...). any suggestions will be gladly considered. oVirt infra team. ----- Original Message -----