- Infra - oVirt List Archives

[JIRA] (OVIRT-2338) resources got stuck
by Dafna Ron (oVirt JIRA) 18 Jul '18

18 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2338?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2338: ---------------------------------- [~bkorren(a)redhat.com] [~ena(a)redhat.com] based on the mail, can we close this jira as resolved and continue follow through the epic? > resources got stuck > ------------------- > > Key: OVIRT-2338 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2338 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Emil Natan > Assignee: infra > Labels: ost_failures, ost_infra > > resources.ovirt.org got stuck. Initially we received different nagios alerts about number of processes and filesystems usage, but the root cause was "Socket timeout after 10 seconds". There was not ssh connectivity, so reset of the VM through the engine UI helped to get it running again. > The issue affected few CQ tests. > Possible improvement could be to set watchdog to automatically reboot the VM if it gets stuck. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

Failure if resources.ovirt.org on Tuesday
by Barak Korren 18 Jul '18

18 Jul '18

We had an issue with resources.ovirt.org becoming unresponsive to all network access on Tuesday, July 17th. Tracking of the issue was done using: https://ovirt-jira.atlassian.net/browse/OVIRT-2338 During the attempt to analyse and fix this issue we came across several other issues the have to do with out ability to gain access to the production VMs. Thos issues hade been documented and collected in the following epic: https://ovirt-jira.atlassian.net/browse/OVIRT-2337 We need to resolve these issues ASAP to ensure that next time such a production outage occurs we don't end up struggling for many minutes to get basic access. In the end we reached the conclusion that the server itself was stuck and needed a hard-reset. This raises the question why didn't we have a watchdog device configured on the VM to automatically detect and deal with such issues. This experience led us to the understanding that resources is fulfilling far too many critical roles ATM for us to be able to responsibly keep it as a simple single VM. I've created the following Epic to track and discuss work for improving the infrastructure behind resources.ovirt.org to make it less fragile and more reliable: https://ovirt-jira.atlassian.net/browse/OVIRT-2344 -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0

[JIRA] (OVIRT-2337) Ensure emergency access to PHX production VMs
by Barak Korren (oVirt JIRA) 18 Jul '18

18 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2337?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-2337: -------------------------------- Summary: Ensure emergency access to PHX production VMs (was: Ensure emergency access to PHC production VMs) > Ensure emergency access to PHX production VMs > --------------------------------------------- > > Key: OVIRT-2337 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2337 > Project: oVirt - virtualization made easy > Issue Type: Epic > Components: oVirt Infra > Reporter: Barak Korren > Assignee: infra > Priority: High > > Make sure that in case of malfunction we have multiple fail-safe ways of gaining access to the PHX production VMs to resolve issues. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2287) ovirt-system-tests_he-node-ng-suite-master is failing on not enough memory to run VMs
by Dafna Ron (oVirt JIRA) 18 Jul '18

18 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2287?page=com.atlassian.jira.… ] Dafna Ron reassigned OVIRT-2287: -------------------------------- Assignee: Gal Ben Haim (was: infra) > ovirt-system-tests_he-node-ng-suite-master is failing on not enough memory to run VMs > ------------------------------------------------------------------------------------- > > Key: OVIRT-2287 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2287 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: sbonazzo > Assignee: Gal Ben Haim > Priority: Highest > Labels: ost_failures, ost_lago > Attachments: srv.log > > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/16… > Cannot run VM. There is no host that satisfies current scheduling > constraints. See below for details:, The host > lago-he-node-ng-suite-master-host-0 did not satisfy internal filter Memory > because its available memory is too low (656 MB) to run the VM. > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2311) ovirt-master vdsm failure - extend disk test times out
by Dafna Ron (oVirt JIRA) 18 Jul '18

18 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2311?page=com.atlassian.jira.… ] Dafna Ron reassigned OVIRT-2311: -------------------------------- Assignee: danken (was: infra) > ovirt-master vdsm failure - extend disk test times out > ------------------------------------------------------- > > Key: OVIRT-2311 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2311 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: danken > Labels: ost_failures > > There is no failure of a test but we can see a gap of an hour from start of extend_disk1 until unhanded exception is thrown. > Although the error is reported on 004_basic_sanity the logs stop on post-003_basic_networking.py. > looking at the available logs it may be a task that is not cleared. > I do not see a connection between the patch and the timeout unless something in the tests is looking ERROR: InterpreterNotFound: python3.6 errors in order to exit. > patch that failed: https://gerrit.ovirt.org/#/c/92847/ > job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8648/ > Error I can see in the logs is: > 2018-07-12 01:23:14,863-04 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [] An error occurred while fetching unregistered disks from Storage Domain id '597bfe1f-d88c-4961-b1dc-f6c771af9 > fff' > but I can't see an error in vdsm that would suggest we had a failure. > you can see the gap in the job: > 05:31:45 [basic-suit] # extend_disk1: > 06:31:56 [basic-suit] * Unhandled exception in <function <lambda> at 0x7f5dd403c5f0> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2344) Re-engineer the oVirt resources server
by Barak Korren (oVirt JIRA) 18 Jul '18

18 Jul '18

Barak Korren created OVIRT-2344: ----------------------------------- Summary: Re-engineer the oVirt resources server Key: OVIRT-2344 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2344 Project: oVirt - virtualization made easy Issue Type: Epic Components: Repositories Mgmt Reporter: Barak Korren Assignee: infra resources.ovirt.org is one of the most important servers in our infrastructure, but its current configuration is quite fragile and prone to fialure. This EPIC is eant to consolidate and track work we do to make resources.ovirt.org more reliable, and maintainable. Among the things we can try to do: # Have multiple redundant front-ends serving package content to users # Separate test and production package serving # Separate package serving and backed repository building # Host resources on 3rd party infrastructure. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2264) Cannot deactivate Storage while there are running tasks on this Storage
by Dafna Ron (oVirt JIRA) 18 Jul '18

18 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2264?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2264: ---------------------------------- https://gerrit.ovirt.org/#/c/93106/ > Cannot deactivate Storage while there are running tasks on this Storage > ----------------------------------------------------------------------- > > Key: OVIRT-2264 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2264 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > we failed test: 007_sd_reattach.deactivate_storage_domain in vdsm project with error: Cannot deactivate Storage while there are running tasks on this Storage. -Please wait until tasks will finish and try again.]". HTTP response code is 409. > I am opening this as a race in case something changed in OST tests which would cause this to repeat. > https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2506 > https://gerrit.ovirt.org/#/c/92313/2 -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2264) Cannot deactivate Storage while there are running tasks on this Storage
by Dafna Ron (oVirt JIRA) 18 Jul '18

18 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2264?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2264: ---------------------------------- This failure also happened in 4.2 I'll submit a patch to skip in 4.2 as well. > Cannot deactivate Storage while there are running tasks on this Storage > ----------------------------------------------------------------------- > > Key: OVIRT-2264 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2264 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > we failed test: 007_sd_reattach.deactivate_storage_domain in vdsm project with error: Cannot deactivate Storage while there are running tasks on this Storage. -Please wait until tasks will finish and try again.]". HTTP response code is 409. > I am opening this as a race in case something changed in OST tests which would cause this to repeat. > https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2506 > https://gerrit.ovirt.org/#/c/92313/2 -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[CQ]: 93052,3 (ovirt-engine) failed "ovirt-4.2" system tests
by oVirt Jenkins 18 Jul '18

18 Jul '18

Change 93052,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93052/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2661/

3 2

[oVirt Jenkins] ovirt-node-ng_ovirt-4.2_build-artifacts-el7-x86_64 - Build # 343 - Failure!
by jenkins＠jenkins.phx.ovirt.org 18 Jul '18

18 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.2_build-artifacts-el7-x8… Build: http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.2_build-artifacts-el7-x8… Build Number: 343 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #343 [Ryan Barry] kickstart: let VDSM reconfigure itself ----------------- Failed Tests: ----------------- No tests ran.

1 1