[JIRA] (OVIRT-2610) Add alerts for Jenkins job queue

Add alerts for Jenkins job queue -------------------------------- Key: OVIRT-2610 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2610 Project: oVirt - virtualization made easy Issue Type: Improvement Reporter: Evgheni Dereveanchin Assignee: Evgheni Dereveanchin We do have some monitoring in place for Jenkins job queues yet there are no alerts configured. Opening ticket to discuss adding alerts and normal actions that should be taken if one happens. Sample actions for the infra owner: * verify that jobs aren't stuck waiting for a label that has no hosts assigned * verify that a stuck job isn't blocking queues (CQ in particular) Monitoring can be added to Nagios using a trivial plugin based on API output: # curl jenkins.ovirt.org/queue/api/xml | grep -o '<item' | wc -l