[ OST Failure Report ] [ oVirt master ] [ 19/06/17 ] [ add_secondary_storage_domains ]
Test failed: 002_bootstrap.add_secondary_storage_domains Link to suspected patches: Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7264 Link to all logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7264/artifa... Error snippet from the log: Host 0: 2017-06-19 08:46:48,457-0400 WARN (jsonrpc/1) [storage.HSM] getPV failed for guid: 360014050f775dd654404bd39d061693c (hsm:1928) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 1925, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014050f775dd654404bd39d061693c',)" 2017-06-19 08:46:48,546-0400 WARN (jsonrpc/1) [storage.LVM] lvm pvs failed: 5 [] [' Failed to find physical volume "/dev/mapper/360014050fae4a3a8e8047e6933531876".'] (lvm:322) 2017-06-19 08:46:48,546-0400 WARN (jsonrpc/1) [storage.HSM] getPV failed for guid: 360014050fae4a3a8e8047e6933531876 (hsm:1928) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 1925, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014050fae4a3a8e8047e6933531876',)" Host 1: *2017-06-19 08:49:03,151-0400 ERROR (monitor/b92f872) [storage.Monitor] Setting up monitor for b92f8727-8c51-4e12-a3fe-5441531c13ca failed (monitor:329) Traceback (most recent call last): File "/usr/share/vdsm/storage/monitor.py", line 326, in _setupLoop self._setupMonitor() File "/usr/share/vdsm/storage/monitor.py", line 348, in _setupMonitor self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 237, in wrapper value = meth(self, *a, **kw) File "/usr/share/vdsm/storage/monitor.py", line 366, in _produceDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 108, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 49, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 132, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 149, in _findDomain return findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'b92f8727-8c51-4e12-a3fe-5441531c13ca',)*
On Mon, Jun 19, 2017 at 4:48 PM, Gil Shinar <gshinar@redhat.com> wrote:
Test failed: 002_bootstrap.add_secondary_storage_domains
Link to suspected patches:
Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7264
Link to all logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7264/artifa...
We have an open bug on this, the mount to the ISO domain gets stuck and never returns: call mount with (u'192.168.201.3:/exports/nfs/iso', u'/rhev/data-center/mnt/192.168.201.3:_exports_nfs_iso') {'vfstype': 'nfs', 'mntOpts': 'soft,nosharecache,timeo=600,retrans=6,nfsvers=3', 'timeout': None, 'cgroup': None} We'll need probably a live system where it reproduces to see where the mount command gets stuck. Sounds like an OS issue to me. BTW, since we are not using the ISO domain, perhaps we should remove its creation. It'll save time. (note however that this is the only NFSv3 mount we perform though). Y.
Error snippet from the log: Host 0:
2017-06-19 08:46:48,457-0400 WARN (jsonrpc/1) [storage.HSM] getPV failed for guid: 360014050f775dd654404bd39d061693c (hsm:1928) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 1925, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014050f775dd654404bd39d061693c',)" 2017-06-19 08:46:48,546-0400 WARN (jsonrpc/1) [storage.LVM] lvm pvs failed: 5 [] [' Failed to find physical volume "/dev/mapper/360014050fae4a3a8e8047e6933531876".'] (lvm:322) 2017-06-19 08:46:48,546-0400 WARN (jsonrpc/1) [storage.HSM] getPV failed for guid: 360014050fae4a3a8e8047e6933531876 (hsm:1928) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 1925, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014050fae4a3a8e8047e6933531876',)"
Host 1:
*2017-06-19 08:49:03,151-0400 ERROR (monitor/b92f872) [storage.Monitor] Setting up monitor for b92f8727-8c51-4e12-a3fe-5441531c13ca failed (monitor:329) Traceback (most recent call last): File "/usr/share/vdsm/storage/monitor.py", line 326, in _setupLoop self._setupMonitor() File "/usr/share/vdsm/storage/monitor.py", line 348, in _setupMonitor self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 237, in wrapper value = meth(self, *a, **kw) File "/usr/share/vdsm/storage/monitor.py", line 366, in _produceDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 108, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 49, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 132, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 149, in _findDomain return findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'b92f8727-8c51-4e12-a3fe-5441531c13ca',)*
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Tue, Jun 20, 2017 at 9:56 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Jun 19, 2017 at 4:48 PM, Gil Shinar <gshinar@redhat.com> wrote:
Test failed: 002_bootstrap.add_secondary_storage_domains
Link to suspected patches:
Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7264
Link to all logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7264/artifa...
We have an open bug on this, the mount to the ISO domain gets stuck and never returns: call mount with (u'192.168.201.3:/exports/nfs/iso', u'/rhev/data-center/mnt/192.168.201.3:_exports_nfs_iso') {'vfstype': 'nfs', 'mntOpts': 'soft,nosharecache,timeo=600,retrans=6,nfsvers=3', 'timeout': None, 'cgroup': None}
We'll need probably a live system where it reproduces to see where the mount command gets stuck. Sounds like an OS issue to me.
Its a bit problematic since its not failing all the time, what we can do is export the ENV once its fails so we can import it offline and debug it. We are planning to release a new Lago version next week with some fixes that hopefully will allow this option.
BTW, since we are not using the ISO domain, perhaps we should remove its creation. It'll save time. (note however that this is the only NFSv3 mount we perform though).
Should we open a bug / ticket on it? who can take ownership on modifying the test?
Y.
Error snippet from the log: Host 0:
2017-06-19 08:46:48,457-0400 WARN (jsonrpc/1) [storage.HSM] getPV failed for guid: 360014050f775dd654404bd39d061693c (hsm:1928) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 1925, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014050f775dd654404bd39d061693c',)" 2017-06-19 08:46:48,546-0400 WARN (jsonrpc/1) [storage.LVM] lvm pvs failed: 5 [] [' Failed to find physical volume "/dev/mapper/360014050fae4a3a8e8047e6933531876".'] (lvm:322) 2017-06-19 08:46:48,546-0400 WARN (jsonrpc/1) [storage.HSM] getPV failed for guid: 360014050fae4a3a8e8047e6933531876 (hsm:1928) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 1925, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014050fae4a3a8e8047e6933531876',)"
Host 1:
*2017-06-19 08:49:03,151-0400 ERROR (monitor/b92f872) [storage.Monitor] Setting up monitor for b92f8727-8c51-4e12-a3fe-5441531c13ca failed (monitor:329) Traceback (most recent call last): File "/usr/share/vdsm/storage/monitor.py", line 326, in _setupLoop self._setupMonitor() File "/usr/share/vdsm/storage/monitor.py", line 348, in _setupMonitor self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 237, in wrapper value = meth(self, *a, **kw) File "/usr/share/vdsm/storage/monitor.py", line 366, in _produceDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 108, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 49, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 132, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 149, in _findDomain return findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'b92f8727-8c51-4e12-a3fe-5441531c13ca',)*
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal edri ASSOCIATE MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Tue, Jun 20, 2017 at 10:28 AM, Eyal Edri <eedri@redhat.com> wrote:
On Tue, Jun 20, 2017 at 9:56 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Jun 19, 2017 at 4:48 PM, Gil Shinar <gshinar@redhat.com> wrote:
Test failed: 002_bootstrap.add_secondary_storage_domains
Link to suspected patches:
Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7264
Link to all logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7264/artifa...
We have an open bug on this, the mount to the ISO domain gets stuck and never returns: call mount with (u'192.168.201.3:/exports/nfs/iso', u'/rhev/data-center/mnt/192.168.201.3:_exports_nfs_iso') {'vfstype': 'nfs', 'mntOpts': 'soft,nosharecache,timeo=600,retrans=6,nfsvers=3', 'timeout': None, 'cgroup': None}
We'll need probably a live system where it reproduces to see where the mount command gets stuck. Sounds like an OS issue to me.
Its a bit problematic since its not failing all the time, what we can do is export the ENV once its fails so we can import it offline and debug it. We are planning to release a new Lago version next week with some fixes that hopefully will allow this option.
BTW, since we are not using the ISO domain, perhaps we should remove its creation. It'll save time. (note however that this is the only NFSv3 mount we perform though).
Should we open a bug / ticket on it? who can take ownership on modifying the test?
There's an open bug about the issue itself. I'm just too lazy to find it. We can open a bug on removing the ISO domain creation, since we don't use it. I'd like us to switch one of the other domains to NFSv3 if possible, though. Y.
Y.
Error snippet from the log: Host 0:
2017-06-19 08:46:48,457-0400 WARN (jsonrpc/1) [storage.HSM] getPV failed for guid: 360014050f775dd654404bd39d061693c (hsm:1928) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 1925, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014050f775dd654404bd39d061693c',)" 2017-06-19 08:46:48,546-0400 WARN (jsonrpc/1) [storage.LVM] lvm pvs failed: 5 [] [' Failed to find physical volume "/dev/mapper/360014050fae4a3a8e8047e6933531876".'] (lvm:322) 2017-06-19 08:46:48,546-0400 WARN (jsonrpc/1) [storage.HSM] getPV failed for guid: 360014050fae4a3a8e8047e6933531876 (hsm:1928) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 1925, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014050fae4a3a8e8047e6933531876',)"
Host 1:
*2017-06-19 08:49:03,151-0400 ERROR (monitor/b92f872) [storage.Monitor] Setting up monitor for b92f8727-8c51-4e12-a3fe-5441531c13ca failed (monitor:329) Traceback (most recent call last): File "/usr/share/vdsm/storage/monitor.py", line 326, in _setupLoop self._setupMonitor() File "/usr/share/vdsm/storage/monitor.py", line 348, in _setupMonitor self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 237, in wrapper value = meth(self, *a, **kw) File "/usr/share/vdsm/storage/monitor.py", line 366, in _produceDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 108, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 49, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 132, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 149, in _findDomain return findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'b92f8727-8c51-4e12-a3fe-5441531c13ca',)*
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
participants (3)
-
Eyal Edri -
Gil Shinar -
Yaniv Kaul