June 2017 - Infra - oVirt List Archives

[JIRA] (OVIRT-1492) Status.ovirt.org doesn't report real status
by eedri (oVirt JIRA) 29 Jun '17

29 Jun '17

eedri created OVIRT-1492: ---------------------------- Summary: Status.ovirt.org doesn't report real status Key: OVIRT-1492 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1492 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: eedri Assignee: infra Some services are not reachable but it is not shown on status page. -- This message was sent by Atlassian JIRA (v1000.1092.0#100053)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el6-x86_64 #495
by jenkins＠jenkins.phx.ovirt.org 29 Jun '17

29 Jun '17

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el6-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el6-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/13/75913/5:patch --prune > git rev-parse origin/patch^{commit} # timeout=10 > git rev-parse patch^{commit} # timeout=10 Checking out Revision 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f (patch) > git config core.sparsecheckout # timeout=10 > git checkout -f 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f > git rev-list 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f # timeout=10 [system-sync_mirrors-centos-updates-el6-x86_64] $ /bin/bash -xe /tmp/hudson6228168351360021776.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el6 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1470, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1248, in _getFileRepoXML size=102400) # setting max size as 100K File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1020, in _getFile result = self.grab.urlgrab(misc.to_utf8(relative), local, File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 699, in <lambda> grab = property(lambda self: self._getgrab()) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 694, in _getgrab self._setupGrab() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 631, in _setupGrab urls = self.urls File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 876, in <lambda> urls = property(fget=lambda self: self._geturls(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 873, in _geturls self._baseurlSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 819, in _baseurlSetup mirrorurls.extend(list(self.metalink_data.urls())) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 916, in <lambda> metalink_data = property(fget=lambda self: self._getMetalink(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 912, in _getMetalink self._metalink = metalink.MetaLinkRepoMD(self.metalink_filename) File "/usr/lib/python2.7/site-packages/yum/metalink.py", line 189, in __init__ raise MetaLinkRepoErrorParseFail, "File %s is not XML" % filename yum.metalink.MetaLinkRepoErrorParseFail: File /home/jenkins/mirrors_cache/fedora-updates-fc24/metalink.xml is not XML Build step 'Execute shell' marked build as failure

1 1

[Lago][HC] tuned service fails to start on one of HC hosts
by Sahina Bose 29 Jun '17

29 Jun '17

One of the host fails to install with Failed to install Host lago-hc-basic-suite-master-host2. Failed to execute stage 'Misc configuration': Failed to start service 'tuned' The installation on other 2 hosts have been successful - all hosts are created from same template/repo. So this is strange. On Wed, Jun 28, 2017 at 8:58 AM, <jenkins(a)jenkins.phx.ovirt.org> wrote: > Project: http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/ > Build: http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/154/ > Build Number: 154 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #154 > [Daniel Belenky] Exclude packages from ovirt-master.repo > > [Barak Korren] Filter builds sent to change queues by version > > [Barak Korren] Make change queue invoke OST basic suit > > [Barak Korren] Various UX improvements in change-queue jobs > > [Barak Korren] Add a job for direct deployments to 'tested' > > [Barak Korren] Use new job to update 'tested' from experimental > > [Barak Korren] Use new tested deploy job in timed builders > > [Eyal Edri] update build retention policy for big artifacts > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: 002_bootstrap.add_hosts > > Error Message: > Host lago-hc-basic-suite-master-host2 failed to install > -------------------- >> begin captured logging << -------------------- > ovirtlago.testlib: ERROR: * Unhandled exception in <function > _host_is_up at 0x3f11cf8> > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, > in assert_equals_within > res = func() > File "/home/jenkins/workspace/ovirt_master_hc-system-tests/ > ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", > line 151, in _host_is_up > raise RuntimeError('Host %s failed to install' % host.name()) > RuntimeError: Host lago-hc-basic-suite-master-host2 failed to install > --------------------- >> end captured logging << --------------------- > > Stack Trace: > File "/usr/lib64/python2.7/unittest/case.py", line 369, in run > testMethod() > File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in > runTest > self.test(*self.arg) > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, > in wrapped_test > test() > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, > in wrapper > return func(get_test_prefix(), *args, **kwargs) > File "/home/jenkins/workspace/ovirt_master_hc-system-tests/ > ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", > line 164, in add_hosts > testlib.assert_true_within(_host_is_up, timeout=15 * 60) > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 256, > in assert_true_within > assert_equals_within(func, True, timeout, allowed_exceptions) > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, > in assert_equals_within > res = func() > File "/home/jenkins/workspace/ovirt_master_hc-system-tests/ > ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", > line 151, in _host_is_up > raise RuntimeError('Host %s failed to install' % host.name()) > Host lago-hc-basic-suite-master-host2 failed to install > -------------------- >> begin captured logging << -------------------- > ovirtlago.testlib: ERROR: * Unhandled exception in <function > _host_is_up at 0x3f11cf8> > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, > in assert_equals_within > res = func() > File "/home/jenkins/workspace/ovirt_master_hc-system-tests/ > ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", > line 151, in _host_is_up > raise RuntimeError('Host %s failed to install' % host.name()) > RuntimeError: Host lago-hc-basic-suite-master-host2 failed to install > --------------------- >> end captured logging << ---------------------

2 3

[JIRA] (OVIRT-1491) Change oVirt DC production storage
by eedri (oVirt JIRA) 29 Jun '17

29 Jun '17

eedri created OVIRT-1491: ---------------------------- Summary: Change oVirt DC production storage Key: OVIRT-1491 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1491 Project: oVirt - virtualization made easy Issue Type: Task Reporter: eedri Assignee: infra Priority: Highest It seems that during the history of the DC, there wasn't one time we really used the DRDB features and it makes our storage configuration more complex and hard to maintain/debug than it brings value. Lets think on a new SIMPLE storage setup that will increase the availability of oVirt services and will prevent incidents like before where we have a single point of failure if the storage is down. without multiple services that adds more failure points and complexity to the setup. One suggestion was to install 2 new storage servers running ISCSI and connect them independently to oVirt and spread the load over both of them, so if one is down the other will keep running. We can consider having a few redundant VMs on both servers if possible like the resources server and maybe others as well. Please add more suggestions on this ticket, it should be our number one priority right now. -- This message was sent by Atlassian JIRA (v1000.1092.0#100053)

1 0

[JIRA] (OVIRT-1491) Change oVirt DC production storage
by eedri (oVirt JIRA) 29 Jun '17

29 Jun '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1491?page=com.atlassian.jira.… ] eedri updated OVIRT-1491: ------------------------- Epic Link: OVIRT-403 > Change oVirt DC production storage > ----------------------------------- > > Key: OVIRT-1491 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1491 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: eedri > Assignee: infra > Priority: Highest > > It seems that during the history of the DC, there wasn't one time we really used the DRDB features and it makes our storage configuration more complex and hard to maintain/debug than it brings value. > Lets think on a new SIMPLE storage setup that will increase the availability of oVirt services and will prevent incidents like before where we have a single point of failure if the storage is down. without multiple services that adds more failure points and complexity to the setup. > One suggestion was to install 2 new storage servers running ISCSI and connect them independently to oVirt and spread the load over both of them, so if one is down the other will keep running. > We can consider having a few redundant VMs on both servers if possible like the resources server and maybe others as well. > Please add more suggestions on this ticket, it should be our number one priority right now. -- This message was sent by Atlassian JIRA (v1000.1092.0#100053)

1 0

[JIRA] (OVIRT-1490) Simplify oVirt sotrage configuration
by Barak Korren (oVirt JIRA) 29 Jun '17

29 Jun '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1490?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-1490: ------------------------------------- [~eedri] [~ederevea] lets start discussion here. > Simplify oVirt sotrage configuration > ------------------------------------ > > Key: OVIRT-1490 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1490 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: storage > Reporter: Barak Korren > Assignee: infra > Priority: Highest > Labels: infra, storage > > Today's outage was a clear reminder that our current storage configuration does not serve us well. We hardly know how to debug it, it seems to not be resistant to the very issues it was supposed to protect against and introduce potential failure scenarios of its own. > I suggest we implement a new storage layout that meets the following criteria: > # Ultimate simplicity at the lower level of the stack. More specifically: > ## The storage severs should be simple NFS or iSCSI servers. No DRBD and no exotic file-systems. > ## Only simple storage will be presented to oVirt for use as storage domains > # Separation of resources between critical services - The 'Jenkins" master for e.g. should not share resources with the "resources" server or anything else.The separation should hold true down to the physical spindle level. > # Duplication of services and use of local storage where possible - this is a longer term effort - but we have some low hanging fruits here like artifactory, where simple DNS/LB-based fail-over between two identical hosts would probably suffice. > # Complexity only where needed and up the stack. For example we can just have the storage for Jenkins be mirrored at the VM level with fail-over to a backup VM. -- This message was sent by Atlassian JIRA (v1000.1092.0#100053)

1 0

[JIRA] (OVIRT-1490) Simplify oVirt sotrage configuration
by Barak Korren (oVirt JIRA) 29 Jun '17

29 Jun '17

Barak Korren created OVIRT-1490: ----------------------------------- Summary: Simplify oVirt sotrage configuration Key: OVIRT-1490 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1490 Project: oVirt - virtualization made easy Issue Type: Improvement Components: storage Reporter: Barak Korren Assignee: infra Priority: Highest Today's outage was a clear reminder that our current storage configuration does not serve us well. We hardly know how to debug it, it seems to not be resistant to the very issues it was supposed to protect against and introduce potential failure scenarios of its own. I suggest we implement a new storage layout that meets the following criteria: # Ultimate simplicity at the lower level of the stack. More specifically: ## The storage severs should be simple NFS or iSCSI servers. No DRBD and no exotic file-systems. ## Only simple storage will be presented to oVirt for use as storage domains # Separation of resources between critical services - The 'Jenkins" master for e.g. should not share resources with the "resources" server or anything else.The separation should hold true down to the physical spindle level. # Duplication of services and use of local storage where possible - this is a longer term effort - but we have some low hanging fruits here like artifactory, where simple DNS/LB-based fail-over between two identical hosts would probably suffice. # Complexity only where needed and up the stack. For example we can just have the storage for Jenkins be mirrored at the VM level with fail-over to a backup VM. -- This message was sent by Atlassian JIRA (v1000.1092.0#100053)

1 0

[JIRA] (OVIRT-1490) Simplify oVirt sotrage configuration
by Barak Korren (oVirt JIRA) 29 Jun '17

29 Jun '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1490?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1490: -------------------------------- Epic Link: OVIRT-403 > Simplify oVirt sotrage configuration > ------------------------------------ > > Key: OVIRT-1490 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1490 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: storage > Reporter: Barak Korren > Assignee: infra > Priority: Highest > Labels: infra, storage > > Today's outage was a clear reminder that our current storage configuration does not serve us well. We hardly know how to debug it, it seems to not be resistant to the very issues it was supposed to protect against and introduce potential failure scenarios of its own. > I suggest we implement a new storage layout that meets the following criteria: > # Ultimate simplicity at the lower level of the stack. More specifically: > ## The storage severs should be simple NFS or iSCSI servers. No DRBD and no exotic file-systems. > ## Only simple storage will be presented to oVirt for use as storage domains > # Separation of resources between critical services - The 'Jenkins" master for e.g. should not share resources with the "resources" server or anything else.The separation should hold true down to the physical spindle level. > # Duplication of services and use of local storage where possible - this is a longer term effort - but we have some low hanging fruits here like artifactory, where simple DNS/LB-based fail-over between two identical hosts would probably suffice. > # Complexity only where needed and up the stack. For example we can just have the storage for Jenkins be mirrored at the VM level with fail-over to a backup VM. -- This message was sent by Atlassian JIRA (v1000.1092.0#100053)

1 0

[JIRA] (OVIRT-1489) refactor git push map file to be regex instead of url
by Gil Shinar (oVirt JIRA) 29 Jun '17

29 Jun '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1489?page=com.atlassian.jira.… ] Gil Shinar reassigned OVIRT-1489: --------------------------------- Assignee: Gil Shinar (was: infra) > refactor git push map file to be regex instead of url > ----------------------------------------------------- > > Key: OVIRT-1489 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1489 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Gil Shinar > Assignee: Gil Shinar > > changing the way the poll script reads stuff from the git push map file. > Instead of using implicit URL, use regex to be able to support in whatever comes in the future -- This message was sent by Atlassian JIRA (v1000.1092.0#100053)

1 0

[JIRA] (OVIRT-1489) refactor git push map file to be regex instead of url
by Gil Shinar (oVirt JIRA) 29 Jun '17

29 Jun '17

Gil Shinar created OVIRT-1489: --------------------------------- Summary: refactor git push map file to be regex instead of url Key: OVIRT-1489 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1489 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Gil Shinar Assignee: infra changing the way the poll script reads stuff from the git push map file. Instead of using implicit URL, use regex to be able to support in whatever comes in the future -- This message was sent by Atlassian JIRA (v1000.1092.0#100053)

1 0